Setting up the Grafana dashboard using the radixnode CLI

Introduction

RadixDLT uses Prometheus and Grafana to provide nodes and gateways with a real-time monitoring and alert system. Prometheus reads from metrics API endpoints of the node and gateway, and stores it as time series data. Grafana reads this time series data and uses it to build monitoring dashboards for displaying node information and alerts in real-time.

Diagram

The Prometheus/Grafana installation has been packaged as part of the radixnode script. This will allow node runners to get a monitoring dashboard up and running by simply running the script and answering a few questions.

Prerequisites

It probably goes without saying that before you can install the node monitoring software, you must have a node and/or gateway up and running. If you haven’t, then run through one of our guides which will show you how to do it. Start by reading the node and gateway overview to work out which service/s to run.

Monitoring can be set up on the same host as an existing setup, or on another host. Monitoring also includes a database, so can cause increased IO contention. For that reason, we’d recommend running it on a separate host, with connections to the other host/s.

Before you start, you must have the radixnode CLI installed on the host you intend to install monitoring.

If running monitoring a separate host, we’d recommend the following specs:

Model vCPU Memory (GB) Storage(GB) Network Bandwidth (Gbps) Operating System

t3.medium

2

4

Provision a gp2 storage volume.

You should initially provision 60 GB of SSD space

Up to 5

Ubuntu 20.04.2.0 LTS (Focal Fossa)

1. Set up the Nginx Metrics password

In order to make sure the node/gateway is secure as possible, the metrics API endpoint is configured with a separate password in the nginx layer.

On each host with a node or gateway, we need to set up the username/password for this endpoint. Run one of the following radixnode commands (depending on your installation type) to set up the endpoint’s authorization.

  • docker

  • systemd

If PostgreSQL is running locally, this command may not respond. In that case, it is advised to stop the deployed software using command radixnode docker stop and then run this command.
radixnode auth set-metrics-password -m DOCKER
  1. If you are running a systemd node then you must first change to the radixdlt user:

sudo su - radixdlt

Then run the below command to setup password

radixnode auth set-metrics-password -m SYSTEMD

2. Set up the monitoring service

This step will be performed on the host you wish to install monitoring.

The monitoring exposes a user dashboard on port 3000 on your server, so make sure that this port is open and available for Grafana to use.

The monitoring setup is split into two commands: . The config command is used to create a config file for monitoring. This allows to persist configuration that you may customise and does not get overwritten during an update. . The install command is used to install the monitoring using the config file.

2.1. Create config file for monitoring setup

A separate config.yaml file is used for monitoring than the one created for CORE and GATEWAY. By default, it lives at ~/monitoring/config.yaml

There are easy commands to configure monitoring with a Core (on the same host as monitoring, or another host), or with a Core and Gateway (on the same host as monitoring, or on a single other host). For other use cases, check out the Advanced setup.

  • Core (same host)

  • Core (other host)

  • Core & Gateway (same host)

  • Core & Gateway (single other host)

  • Advanced

radixnode monitoring config \
    -m MONITOR_CORE \    (1)
    -cm [metrics_password]  (2)
1 Setup mode selected here as MONITOR_CORE. This only setups config for monitoring core node.
2 The password for the metrics user that was configured in the previous step.

The username for the metrics user is typically metrics, but can be set to something else if the host was configured using the DETAILED config mode.

NODE_HOST_IP_OR_NAME=<host ip or dns name>:<port> \  (1)
radixnode monitoring config \
    -m MONITOR_CORE \    (2)
    -cm [metrics_password]  (3)
1 The host of the Core, and the optional :port to its metrics endpoint (if nonstandard)
2 Setup mode selected here as MONITOR_CORE. This only setups config for monitoring core node.
3 The password for the metrics user that was configured in the previous step.

The username for the metrics user is typically metrics, but can be set to something else if the host was configured using the DETAILED config mode.

radixnode monitoring config \
    -m MONITOR_CORE MONITOR_GATEWAY \    (1)
    -cm [metrics_password]  \ (2)
    -gm [metrics_password]  \ (3)
    -am [metrics_password]  (4)
1 The setup mode is selected here as MONITOR_CORE and MONITOR_GATEWAY. This creates config for monitoring both the core full node and gateway.
2 The password for the metrics user that was configured in the previous step. This is for the core full node where nginx protects path /prometheus/metrics.
3 The password for the metrics user that was configured in the previous step. This is for the network gateway’s gateway api where nginx protects path /gateway/metrics.
4 The password for the metrics user that was configured in the previous step. This is for the network gateway’s data aggregator where nginx protects path /aggregator/metrics.

The parameters cm,gm,am are separate, although in many cases they will share the same password. The options are separate to allow running the services on different hosts, where separate passwords may be used.

The username for the metrics user is typically metrics, but can be set to something else if the host was configured using the DETAILED config mode. In that case, see the Advanced setup.

NODE_HOST_IP_OR_NAME=<host ip or dns name>:<port> \  (1)
radixnode monitoring config \
    -m MONITOR_CORE MONITOR_GATEWAY \    (2)
    -cm [metrics_password]  \ (3)
    -gm [metrics_password]  \ (4)
    -am [metrics_password]  (5)
1 The host of the Core and Gateway, and the optional :port to its metrics endpoints (if nonstandard)
2 The setup mode is selected here as MONITOR_CORE and MONITOR_GATEWAY. This creates config for monitoring both the core full node and gateway.
3 The password for the metrics user that was configured in the previous step. This is for the core full node where nginx protects path /prometheus/metrics.
4 The password for the metrics user that was configured in the previous step. This is for the network gateway’s gateway api where nginx protects path /gateway/metrics.
5 The password for the metrics user that was configured in the previous step. This is for the network gateway’s data aggregator where nginx protects path /aggregator/metrics.

The parameters cm,gm,am are separate, although in many cases they will share the same password. The options are separate to allow running the services on different hosts, where separate passwords may be used.

The username for the metrics user is typically metrics, but can be set to something else if the host was configured using the DETAILED config mode. In that case, see the Advanced setup.

radixnode monitoring config \
    -m DETAILED (1)
1 Setup mode selected here as DETAILED. This will walkthrough you with a series of questions to setup config file appropriately.

Alternatively you can manually edit the config file that is created from other commands to point to the right values of the monitored host.

2.2. Run setup to install and run monitoring

When the monitoring has been configured, you can now run the script to set up monitoring:

radixnode monitoring install

And that’s pretty much it.

3. Accessing the dashboard.

From the monitoring host, you can view your node’s dashboard remotely using any browser, using this URL pattern:

http://<monitoring-host-ip>:3000/d/radix_node_dashboard/radix-node-dashboard?orgId=1&refresh=5s

The monitoring-host-ip is the external IP address of your host where monitoring is running.

Grafana will display a page asking for your username and password.

log into grafana
the Grafana welcome screen

Since this is the first time you’ve run the monitor, enter admin for the username and admin again for the password. Grafana will now display another dialog asking you to change the password for the admin user.

grafana change password
pick a new password

You will now see a blank page with a somewhat discouraging Not Found message at the top. Click the Search icon (the magnifying glass).

opening grafana screen
opening window

Navigate to Dashboards  Radix Node Dashboard:

select dashboard
selecting the dashboard

Grafana will now the example dashboard:

radix node validator dashboard
The Grafana dashboard

4. Accessing the metrics endpoint

Now that the dashboard is up and running, you may want to add new elements yourself, or use the data from the node to build other applications for monitoring and gathering information.

The same data that Grafana uses to build its screen is available on the metrics endpoint/s:

# Core node:
curl -X -k GET --location "https://CORE_NODE_HOST:METRICS_PORT/prometheus/metrics" -H "Content-Type: application/json" --basic --user metrics:nginx-password

# Network Gateway - Data Aggregator
curl -X -k GET --location "https://DATA_AGGREGATOR_HOST:METRICS_PORT/aggregator/metrics" -H "Content-Type: application/json" --basic --user metrics:nginx-password

# Network Gateway - Gateway API
curl -X -k GET --location "https://GATEWAY_API_HOST:METRICS_PORT/gateway/metrics" -H "Content-Type: application/json" --basic --user metrics:nginx-password

Where nginx-password is the password you set up in Section 1, “Set up the Nginx Metrics password”.

5. Shut down Node monitoring

To shut down the monitor on the monitoring host, use the following command:

radixnode monitoring stop