Troubleshooting Problems with your Node

Introduction

If your node has failed to start up, or you’re not getting anything back from your API calls, then you’ve come to the right place. We’ll keep this page updated with problem solutions suggested by our developers and members of the Radix community.

Server not found error after installing node

After installing the node, a terminal call to get the node information:

  • radixnode

  • curl

radixnode api account get-info
curl -d '{ "jsonrpc": "2.0", "method": "account.get_info", "params": [], "id":1}' -H "Content-Type: application/json" -X POST -u superadmin:nginx-password -k  "https://localhost/account"

returns a block of HTML stating that the server hasn’t been found:

<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>

If you find yourself in this situation then there are a number of things you can try:

Restart the Node and Nginx

If this is the first time you’ve tried to run the node after installing it, then it’s possible that nginx and the node are not in sync; the first thing to try is restarting them:

  • docker

  • systemd

To stop the node:
docker-compose -f radix-fullnode-compose.yml down
To start the node:
RADIXDLT_NETWORK_NODE=radix://rdx1qthu8yn06k75dnwpkysyl8smtwn0v4xy29auzjlcrw7vgduxvnwnst6derj@99.80.126.26  RADIXDLT_NODE_KEY_PASSWORD=node-password docker-compose -f radix-fullnode-compose.yml up -d
To restart the node:
sudo su - radixdlt
sudo systemctl restart radixdlt-node.service
sudo systemctl restart nginx

Ensure the IP addresses are correct

A systemd node requires the address of your host IP along with the address of a node to connect to. These values are defined in the /etc/radixdlt/node/default.config file. Make sure that the addresses for host.ip and network.seeds are both correct:

host.ip

must point to the external address of your host server.

If you’re running your node under a Docker instance then the host.ip is automatically configured when you run the compose script.
network.seeds

points to the address of a node you’re connecting to join the network.

Also make sure that you’re using the correct ports.

You can find more details on how these should be set up in the systemd installation guide. Details on setting up the network.seeds address can be found here. The Docker installation guide explains how to set up the network.seeds address for your Docker installation.

Make sure the ports are open

This applies to both the Docker and systemd installations. Check that ports 30000 and 443 are open and available for use (i.e. make sure no other application is using them).

Instructions for configuring ports is given in the guides for installing Docker instance and systemd nodes.

Examine the logs

If the restart didn’t work then the log files are your next stop. The logs can be found in the \etc\radixdlt\node\logs on a systemd installation. They’re archived at the end of each day, but the current log is always named radixdlt-core.log.

If you’re running the node as a Docker instance, then you’ll need to log into the instance before you can access the logs.

If the node is running but not actually communicating, then your log file might already be very large. In this case, it’s best to delete the existing log:

rm /etc/radixdlt/node/logs/radixdlt-core.log

You can use output from the logs to troubleshoot the problem further or get help on the Discord channel: posting relevant sections of the log will help Radix staff and community members towards a possible solution.

“Not enough balance for fee burn” during validator registration

This error is thrown when your validator doesn’t have enough XRD tokens for validator registration.

Make sure your node has enough xrd for validator registration

Fetch your node address by running below the following command:

  • radixnode

  • curl

radixnode api account get-info
curl -d '{ "jsonrpc": "2.0", "method": "account.get_info", "params": [], "id":1}' -H "Content-Type: application/json" -X POST -u superadmin:nginx-password -k  "https://localhost/account"

This should return basic details about your node:

{"result":{"address":"rdx1qspl7mgjqwgwqyjvy2tj8swe8a4lr6mxqdhwmn60cujl6a85mqh69eg37p9ph","balance":{"stakes":[],"tokens":[]}},"id":1,"jsonrpc":"2.0"}

If the node shows a zero balance for xrd, then send some tokens from your wallet to the node.

If the node still shows a zero balance, then it is likely that your node isn’t syncing correctly.

Check the Node syncing

The node may not be syncing for a number of reasons. If you think node is running fine, then you can check its current state version by running command:

  • radixnode

  • curl

radixnode api health
curl -k -u admin:nginx-password "https://localhost/health"

If the node is syncing correctly, then you should see a status code of UP returned in the response message:

{"status":"UP"}

Ensure that port 30000 is open

This can be tested from outside your instance by running below command

telnet <your-node-ip> 30000

The result from the command should indicate that the port is open.

Trying 35.178.70.50...
Connected to 35.178.70.50.
Escape character is '^]'.

If you don’t get a response back then you will need to open port 30000. (You may need to consult your systems administrator or cloud provider to do this)

Once the port is open, try restarting the nodes, then submitting your validator registration again.

Getting 401 errors returned api calls

If you execute the following terminal command:

  • radixnode

  • curl

radixnode api account get-info
curl -d '{ "jsonrpc": "2.0", "method": "account.get_info", "params": [], "id":1}' -H "Content-Type: application/json" -X POST -u superadmin:nginx-password -k  "https://localhost/account"

and receive a 401 error, this means that the password for Nginx is incorrect. If you cannot remember the password then you can reset it from the terminal.

  • docker

  • systemd

docker run --rm -v radixdlt_nginx_secrets:/secrets radixdlt/htpasswd:v1.0.0
htpasswd -bc /secrets/htpasswd.admin admin nginx-password
sudo htpasswd -c /etc/nginx/secrets/htpasswd.admin admin

PKCS12 key store mac invalid - wrong password or corrupted file

This error occurs because the node is unable to read the keyfile. Try the following:

Check the password

For Docker setup, the password is set using environment variable RADIXDLT_NODE_KEY_PASSWORD when you issue the command to restart the node. Check the password is correct, then restart the node.

Check the Keystore file doesn’t have right permission

Set the right key permission by running command:

sudo chmod 644 node-keystore.ks

Out of memory issues

You see this issue in the node container or process logs. This could be either that the heap is not sufficient or there is a memory leak in the version of the software you’re running.

Increase the heap size to 3 GB

  1. Bring the node down by using below command

    docker-compose -f radix-fullnode-compose.yml down
  2. Update the JAVA_OPTS in radix-fullnode-compose.yml: change the setting from -Xms2g -Xmx2g to -Xms8g -Xmx8g

  3. Restart the node with the following command:

    RADIXDLT_NETWORK_NODE=52.48.95.182  RADIXDLT_NODE_KEY_PASSWORD=node-password docker-compose -f radix-fullnode-compose.yml up -d

If the heap is right, then you may need to update the node software.

Excessive Disk Writes to the Radix Ledger

This has been reported by a number of users on the Discord channel, along with a solution, also from a user on the same channel.

Problem

The Berkeley Database (which is the underlying database engine used for the Radix engine) writes a log of database transactions, which are those logged transactions to the database at regular intervals. These intervals (or checkpoints) are relatively expensive operations as they involve rebuilding the database indexes. By default, the checkpoint operation is performed every 20 MB worth of transactional data. So for something like the Radix engine, that’ll be a sizeable number of write operations.

The Solution

is to increase the size limit of the transaction log data that will trigger a checkpoint.

  1. Create a new file called je.properties in the /data directory where your Radix ledger is stored.

  2. Copy the following text into the file:

    # Set the log file size to 1Gb each (Default: 100Mb)
    je.log.fileMax=1073741824
    # Run the checkpointer every ~250Mb of data (Default: 20Mb)
    je.checkpointer.bytesInterval=250000000

    The checkpointer will now be triggered for a larger data set, which decreases the number of reindexing operations the database will perform.

  3. Save the file

  4. Restart your node. (You can find the restart instructions in Restart the Node and Nginx)

Credits

Thanks to Stuart | RadixPool #0282 for his efforts in tracking down and solving this problem.