Troubleshooting Problems with your Node
Introduction
If your node has failed to start up, or you’re not getting anything back from your API calls, then you’ve come to the right place. We’ll keep this page updated with problem solutions suggested by our developers and members of the Radix community.
Server not found error after installing node
After installing the node, a terminal call to get the node information:
radixnode api core network-configuration
curl -k -u admin:nginx-admin-password --request X 'https://localhost/network/configuration' \
--header 'Content-Type: application/json' \
--header 'Authorization: Basic YWRtaW46YWRtaW4=' \
--data-raw '{
}'
returns a block of HTML stating that the server hasn’t been found:
<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>
If you find yourself in this situation then there are a number of things you can try:
Restart the Node and Nginx
If this is the first time you’ve tried to run the node after installing it, then it’s possible that nginx
and the node are not in sync; the first thing to try is restarting them:
radixnode docker stop -f radix-fullnode-compose.yml
radixnode docker start -f radix-fullnode-compose.yml -t radix://rn1qgf0tug4nmxfa7su8zsu8pejzq48eeglvxf8le09cuy0nsghzg44weacz2q@3.109.36.249
radixnode systemd stop
radixnode systemd restart
docker-compose -f radix-fullnode-compose.yml down
RADIXDLT_NETWORK_NODE=radix://rn1qgf0tug4nmxfa7su8zsu8pejzq48eeglvxf8le09cuy0nsghzg44weacz2q@3.109.36.249 RADIXDLT_NODE_KEY_PASSWORD=node-password docker-compose -f radix-fullnode-compose.yml up -d
sudo su - radixdlt
sudo systemctl restart radixdlt-node.service
sudo systemctl restart nginx
Ensure the IP addresses are correct
A systemd node requires the address of your host IP along with the address of a node to connect to. These values are defined in the /etc/radixdlt/node/default.config
file. Make sure that the addresses for host.ip
and network.seeds
are both correct:
host.ip
|
must point to the external address of your host server.
|
||
network.seeds
|
points to the address of a node you’re connecting to join the network. |
Also make sure that you’re using the correct ports.
You can find more details on how these should be set up in the systemd installation guide. Details on setting up the network.seeds
address can be found here. The Docker installation guide explains how to set up the network.seeds
address for your Docker
installation.
Make sure the ports are open
This applies to both the Docker and systemd installations. Check that ports 30000
and 443
are open and available for use (i.e. make sure no other application is using them).
Instructions for configuring ports is given in the guides for installing Docker instance and systemd nodes.
Examine the logs
If the restart didn’t work then the log files are your next stop. The logs can be found in the \etc\radixdlt\node\logs
on a systemd installation. They’re archived at the end of each day, but the current log is always named radixdlt-core.log
.
If you’re running the node as a Docker instance, then you’ll need to log into the instance before you can access the logs. |
If the node is running but not actually communicating, then your log file might already be very large. In this case, it’s best to delete the existing log:
rm /etc/radixdlt/node/logs/radixdlt-core.log
Then restart the node.
You can use output from the logs to troubleshoot the problem further or get help on the Discord channel: posting relevant sections of the log will help Radix staff and community members towards a possible solution.
“Not enough balance for fee burn” during validator registration
This error is thrown when your validator doesn’t have enough XRD
tokens for validator registration.
Make sure your node has enough xrd
for validator registration
Fetch your node address by running below the following command:
radixnode api core entity -a
curl -k -u admin:{nginx-admin-password} -X POST 'https://localhost/entity' \
--header 'Content-Type: application/json' \
--data-raw '{
"network_identifier": {
"network": "<network>"
},
"entity_identifier": {
"address": "<node's wallet address>"
}
}'
This should return basic details about your node:
{
"balances":[],
"data_objects":[
],
"state_identifier":{
"state_version":45937223,
"transaction_accumulator":"147ae7f0edfdf9df43213e6affaf43986a0f0e1abda3f31483d8b1f4db586f97"
}
}
If the node shows a zero balance for xrd
, then send some tokens from your wallet to the node.
If the node still shows a zero balance, then it is likely that your node isn’t syncing correctly.
Check the Node syncing
The node may not be syncing for a number of reasons. If you think node is running fine, then you can check its current state version by running command:
radixnode api system health
curl -k -u admin:nginx-password "https://localhost/system/health"
If the node is syncing correctly, then you should see a status code of UP
returned in the response message:
{"status":"UP"}
Ensure that port 30000 is open
This can be tested from outside your instance by running below command
telnet <your-node-ip> 30000
The result from the command should indicate that the port is open.
Trying 35.178.70.50...
Connected to 35.178.70.50.
Escape character is '^]'.
If you don’t get a response back then you will need to open port 30000. (You may need to consult your systems administrator or cloud provider to do this)
Once the port is open, try restarting the nodes, then submitting your validator registration again.
Getting 401 errors returned api calls
If you execute the following terminal command:
radixnode api core entity -a
curl -k -u admin:{nginx-admin-password} -X POST 'https://localhost/entity' \
--header 'Content-Type: application/json' \
--data-raw '{
"network_identifier": {
"network": "<network>"
},
"entity_identifier": {
"address": "<node's wallet address>"
}
}'
and receive a 401
error, this means that the password for Nginx
is incorrect. If you cannot remember the password then you can reset it from the terminal.
For docker mode
radixnode auth set-admin-password -m DOCKER
For systemd mode
radixnode auth set-admin-password -m SYSTEMD
docker run --rm -v radixdlt_nginx_secrets:/secrets radixdlt/htpasswd:v1.0.0
htpasswd -bc /secrets/htpasswd.admin admin nginx-password
sudo htpasswd -c /etc/nginx/secrets/htpasswd.admin admin
PKCS12 key store mac invalid - wrong password or corrupted file
This error occurs because the node is unable to read the keyfile. Try the following:
Check the password
For Docker setup, the password is set using environment variable RADIXDLT_NODE_KEY_PASSWORD
when you issue the command to restart the node. Check the password is correct, then restart the node.
Out of memory issues
You see this issue in the node container or process logs. This could be either that the heap is not sufficient or there is a memory leak in the version of the software you’re running.
Increase the heap size to 3 GB
-
Bring the node down by using below command
docker-compose -f radix-fullnode-compose.yml down
-
Update the
JAVA_OPTS
inradix-fullnode-compose.yml
: change the setting from-Xms2g -Xmx2g
to-Xms8g -Xmx8g
-
Restart the node with the following command:
RADIXDLT_NETWORK_NODE=52.48.95.182 RADIXDLT_NODE_KEY_PASSWORD=node-password docker-compose -f radix-fullnode-compose.yml up -d
If the heap is right, then you may need to update the node software.
Excessive Disk Writes to the Radix Ledger
This has been reported by a number of users on the Discord channel, along with a solution, also from a user on the same channel.
Problem
The Berkeley Database (which is the underlying database engine used for the Radix engine) writes a log of database transactions, which are those logged transactions to the database at regular intervals. These intervals (or checkpoints) are relatively expensive operations as they involve rebuilding the database indexes. By default, the checkpoint operation is performed every 20 MB worth of transactional data. So for something like the Radix engine, that’ll be a sizeable number of write operations.
The Solution
is to increase the size limit of the transaction log data that will trigger a checkpoint.
-
Create a new file called
je.properties
in the/data
directory where your Radix ledger is stored. -
Copy the following text into the file:
# Set the log file size to 1Gb each (Default: 100Mb) je.log.fileMax=1073741824 # Run the checkpointer every ~250Mb of data (Default: 20Mb) je.checkpointer.bytesInterval=250000000
The checkpointer will now be triggered for a larger data set, which decreases the number of reindexing operations the database will perform.
-
Save the file
-
Restart your node. (You can find the restart instructions in Restart the Node and Nginx)
TooLongFrameException error
From version 1.1.0 onwards, the core node software, supports for nginx to pass originating IP address on tcp connection. This feature requires enabling the proxy protocol in both nginx and core config. Otherwise , if it is enabled only on node and not in nginx, core node will throw below exception
Jan 19 21:25:43 ip-172-31-45-167 radixdlt[1662434]: io.netty.handler.codec.TooLongFrameException: frame length (526) exceeds the allowed maximum (255)
Jan 19 21:25:43 ip-172-31-45-167 radixdlt[1662434]: at io.netty.handler.codec.LineBasedFrameDecoder.fail(LineBasedFrameDecoder.java:158) ~[netty-all-4.1.66.Final.jar:4.1.66.Final]....
To get rid of the above exception, settings for proxy protocol needs to be enabled on both nginx and core
-
For a docker node, refer this docker section. The environment variable
RADIXDLT_NETWORK_USE_PROXY_PROTOCOL
is supposed to be set to true on both nginx and core container -
For systemD node, in
default.config
file , the config value fornetwork.p2p.use_proxy_protocol
should be set to true. In addition, innginx.conf
file, add a line to enable proxy protocol in theserver
block ofstream
server as shown below
stream {
resolver "127.0.0.53";
resolver_timeout 600s;
server {
listen 30000;
proxy_pass 127.0.0.1:30001;
proxy_protocol on;
}
}