Troubleshooting Problems with your Node

Introduction

If your node has failed to start up, or you’re not getting anything back from your API calls, then you’ve come to the right place. We’ll keep this page updated with problem solutions suggested by our developers and members of the Radix community.

Server not found error after installing node

After installing the node, a terminal call to get the node information:

radixnode
curl

radixnode api core network-configuration

curl -k -u admin:nginx-admin-password --request X 'https://localhost/network/configuration' \
--header 'Content-Type: application/json' \
--header 'Authorization: Basic YWRtaW46YWRtaW4=' \
--data-raw '{
}'

returns a block of HTML stating that the server hasn’t been found:

<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>

If you find yourself in this situation then there are a number of things you can try:

Restart the Node and Nginx

If this is the first time you’ve tried to run the node after installing it, then it’s possible that nginx and the node are not in sync; the first thing to try is restarting them:

radixnode
docker
systemd

To stop the docker node using cli:

radixnode docker stop -f radix-fullnode-compose.yml

To start the docker node using cli:

radixnode docker start -f radix-fullnode-compose.yml -t radix://rn1qgf0tug4nmxfa7su8zsu8pejzq48eeglvxf8le09cuy0nsghzg44weacz2q@3.109.36.249

To stop the systemd node using cli:

radixnode systemd stop

To start the systemd node using cli:

radixnode systemd restart

To stop the node:

docker-compose -f radix-fullnode-compose.yml down

To start the node:

RADIXDLT_NETWORK_NODE=radix://rn1qgf0tug4nmxfa7su8zsu8pejzq48eeglvxf8le09cuy0nsghzg44weacz2q@3.109.36.249  RADIXDLT_NODE_KEY_PASSWORD=node-password docker-compose -f radix-fullnode-compose.yml up -d

To restart the node:

sudo su - radixdlt
sudo systemctl restart radixdlt-node.service
sudo systemctl restart nginx

Ensure the IP addresses are correct

A systemd node requires the address of your host IP along with the address of a node to connect to. These values are defined in the /etc/radixdlt/node/default.config file. Make sure that the addresses for host.ip and network.seeds are both correct:

host.ip

must point to the external address of your host server.

If you’re running your node under a Docker instance then the host.ip is automatically configured when you run the compose script.

network.seeds

points to the address of a node you’re connecting to join the network.

Also make sure that you’re using the correct ports.

You can find more details on how these should be set up in the systemd installation guide. Details on setting up the network.seeds address can be found here. The Docker installation guide explains how to set up the network.seeds address for your Docker installation.

Make sure the ports are open

This applies to both the Docker and systemd installations. Check that ports 30000 and 443 are open and available for use (i.e. make sure no other application is using them).

Instructions for configuring ports is given in the guides for installing Docker instance and systemd nodes.

Examine the logs

If the restart didn’t work then the log files are your next stop. The logs can be found in the \etc\radixdlt\node\logs on a systemd installation. They’re archived at the end of each day, but the current log is always named radixdlt-core.log.

If you’re running the node as a Docker instance, then you’ll need to log into the instance before you can access the logs.

If the node is running but not actually communicating, then your log file might already be very large. In this case, it’s best to delete the existing log:

rm /etc/radixdlt/node/logs/radixdlt-core.log

Then restart the node.

You can use output from the logs to troubleshoot the problem further or get help on the Discord channel: posting relevant sections of the log will help Radix staff and community members towards a possible solution.

“Not enough balance for fee burn” during validator registration

This error is thrown when your validator doesn’t have enough XRD tokens for validator registration.

Make sure your node has enough `xrd` for validator registration

Fetch your node address by running below the following command:

radixnode
curl

radixnode api core entity -a

curl -k -u admin:{nginx-admin-password} -X POST 'https://localhost/entity' \
--header 'Content-Type: application/json' \
--data-raw '{
    "network_identifier": {
        "network": "<network>"
    },
    "entity_identifier": {
        "address": "<node's wallet address>"
    }
}'

This should return basic details about your node:

{
   "balances":[],
   "data_objects":[

   ],
   "state_identifier":{
      "state_version":45937223,
      "transaction_accumulator":"147ae7f0edfdf9df43213e6affaf43986a0f0e1abda3f31483d8b1f4db586f97"
   }
}

If the node shows a zero balance for xrd, then send some tokens from your wallet to the node.

If the node still shows a zero balance, then it is likely that your node isn’t syncing correctly.

Check the Node syncing

The node may not be syncing for a number of reasons. If you think node is running fine, then you can check its current state version by running command:

radixnode
curl

radixnode api system health

curl -k -u admin:nginx-password "https://localhost/system/health"

If the node is syncing correctly, then you should see a status code of UP returned in the response message:

{"status":"UP"}

Ensure that port 30000 is open

This can be tested from outside your instance by running below command

telnet <your-node-ip> 30000

The result from the command should indicate that the port is open.

Trying 35.178.70.50...
Connected to 35.178.70.50.
Escape character is '^]'.

If you don’t get a response back then you will need to open port 30000. (You may need to consult your systems administrator or cloud provider to do this)

Once the port is open, try restarting the nodes, then submitting your validator registration again.

Getting 401 errors returned api calls

If you execute the following terminal command:

radixnode
curl

radixnode api core entity -a

curl -k -u admin:{nginx-admin-password} -X POST 'https://localhost/entity' \
--header 'Content-Type: application/json' \
--data-raw '{
    "network_identifier": {
        "network": "<network>"
    },
    "entity_identifier": {
        "address": "<node's wallet address>"
    }
}'

and receive a 401 error, this means that the password for Nginx is incorrect. If you cannot remember the password then you can reset it from the terminal.

radixnode
docker
systemd

For docker mode

radixnode auth set-admin-password -m DOCKER

For systemd mode

radixnode auth set-admin-password -m SYSTEMD

docker run --rm -v radixdlt_nginx_secrets:/secrets radixdlt/htpasswd:v1.0.0
htpasswd -bc /secrets/htpasswd.admin admin nginx-password

sudo htpasswd -c /etc/nginx/secrets/htpasswd.admin admin

PKCS12 key store mac invalid - wrong password or corrupted file

This error occurs because the node is unable to read the keyfile. Try the following:

Check the password

For Docker setup, the password is set using environment variable RADIXDLT_NODE_KEY_PASSWORD when you issue the command to restart the node. Check the password is correct, then restart the node.

Check the Keystore file doesn’t have right permission

Set the right key permission by running command:

sudo chmod 644 node-keystore.ks

Out of memory issues

You see this issue in the node container or process logs. This could be either that the heap is not sufficient or there is a memory leak in the version of the software you’re running.

Increase the heap size to 3 GB

Bring the node down by using below command

docker-compose -f radix-fullnode-compose.yml down

Update the JAVA_OPTS in radix-fullnode-compose.yml: change the setting from -Xms2g -Xmx2g to -Xms8g -Xmx8g

Restart the node with the following command:

RADIXDLT_NETWORK_NODE=52.48.95.182  RADIXDLT_NODE_KEY_PASSWORD=node-password docker-compose -f radix-fullnode-compose.yml up -d

If the heap is right, then you may need to update the node software.

Excessive Disk Writes to the Radix Ledger

This has been reported by a number of users on the Discord channel, along with a solution, also from a user on the same channel.

Problem

The Berkeley Database (which is the underlying database engine used for the Radix engine) writes a log of database transactions, which are those logged transactions to the database at regular intervals. These intervals (or checkpoints) are relatively expensive operations as they involve rebuilding the database indexes. By default, the checkpoint operation is performed every 20 MB worth of transactional data. So for something like the Radix engine, that’ll be a sizeable number of write operations.

The Solution

is to increase the size limit of the transaction log data that will trigger a checkpoint.

Create a new file called je.properties in the /data directory where your Radix ledger is stored.

Copy the following text into the file:

# Set the log file size to 1Gb each (Default: 100Mb)
je.log.fileMax=1073741824
# Run the checkpointer every ~250Mb of data (Default: 20Mb)
je.checkpointer.bytesInterval=250000000

The checkpointer will now be triggered for a larger data set, which decreases the number of reindexing operations the database will perform.

Save the file
Restart your node. (You can find the restart instructions in Restart the Node and Nginx)

Credits

Thanks to Stuart | RadixPool #0282 for his efforts in tracking down and solving this problem.

TooLongFrameException error

From version 1.1.0 onwards, the core node software, supports for nginx to pass originating IP address on tcp connection. This feature requires enabling the proxy protocol in both nginx and core config. Otherwise , if it is enabled only on node and not in nginx, core node will throw below exception

Jan 19 21:25:43 ip-172-31-45-167 radixdlt[1662434]: io.netty.handler.codec.TooLongFrameException: frame length (526) exceeds the allowed maximum (255)
Jan 19 21:25:43 ip-172-31-45-167 radixdlt[1662434]:         at io.netty.handler.codec.LineBasedFrameDecoder.fail(LineBasedFrameDecoder.java:158) ~[netty-all-4.1.66.Final.jar:4.1.66.Final]....

To get rid of the above exception, settings for proxy protocol needs to be enabled on both nginx and core

For a docker node, refer this docker section. The environment variable RADIXDLT_NETWORK_USE_PROXY_PROTOCOL is supposed to be set to true on both nginx and core container
For systemD node, in default.config file , the config value for network.p2p.use_proxy_protocol should be set to true. In addition, in nginx.conf file, add a line to enable proxy protocol in the server block of stream server as shown below

stream {
   resolver "127.0.0.53";
   resolver_timeout 600s;
   server {
     listen 30000;
     proxy_pass 127.0.0.1:30001;
     proxy_protocol on;
   }
}