# Welcome to Avaunt Staking’s Radix Validator Guide

## Introduction <a href="#introduction" id="introduction"></a>

This guide is intended for potential Radix node runners who would like to run their own node on the Radix DLT network. The guide will walk you through the steps to build a Node Server and an optional monitoring Webserver using Prometheus and Grafana.&#x20;

To learn more about Radix and the wonderful things they are building visit the Radix DLT site [here](https://www.radixdlt.com/)

**Notes**:

* The steps included are based on hosting via Amazon Web Services (AWS) but could be transferred to another cloud or physical infrastructure environment.
* This guide is based on running software and commands from a Windows Desktop. Some steps may be different on a Linux or MAC OS.
* This guide demonstrates one of the many available configurable options for building and running a node and will be based on [Docker.](https://www.docker.com/) There are alternative strategies available and you should research the pros and cons for each setup.
* There is a dependency on ensuring you have configured your AWS account, base AWS infrastructure and security groups as required.

## Pre-requisites

1. An AWS account and understanding of how to build instances, configure security groups and firewall rules in the EC2 Dashboard.
2. Putty and PuttyGen installed on your computer
3. A basic understanding of Linux commands
4. An understanding of Docker

## Contents <a href="#contents" id="contents"></a>

1. Pre-requisites
2. High-level Build Configuration
3. Configure the Node Server
4. Install Node Exporter
5. Configuring the webserver with Prometheus & Grafana
6. Alerting & Monitoring
7. Optional Configuration
8. Add the Radix Node Dashboard to Grafana


# High-Level Build Configuration

## Introduction

{% hint style="danger" %}
This guide won't cover topics such as high availability, failover and resilience. If or how you build resilience into your design is a complex topic and there are many different ways it could be achieved.  There is also a risk that including processes like automatic failover could cause double-signing of proposals which could at a future date lead to slashing.&#x20;
{% endhint %}

We will be configuring two servers. In both cases there is an assumption you have already built the servers and are starting at a base Linux image level. There are plenty of guides and tutorials on the Internet on how to build and correctly configure an EC2 Instance which I won't provide here.

1. The Radix Validator Node Server
2. A Web Server for hosting Prometheus and Grafana

### Node Server Hardware (AWS EC2)

* c5.2xlarge
* 8 vCPU
* 16GB memory
* 100GB of GP2 storage increasing in size as required
* 10Gbps of Network bandwidth
* Ubuntu 20.0.4.2.0 LTS (Focal Fossa)

### Node Server Software

* Node exporter
* fail2ban
* Google authenticator 2FA

### Web Server Hardware (AWS EC2)

* t2.micro (AWS free-tier)
* 1 vCPU
* 1GB memory
* 8GB storage (magnetic will be fine)&#x20;

### Web Server Software

* Grafana
* Prometheus
* Nginx Web Server
* Let’s Encrypt

## &#x20;<a href="#configure-the-node-server" id="configure-the-node-server"></a>


# Pre-configure the Node Server

The steps to prepare and secure your validating node server prior to installing the Radix Node Software.

### Connect to the server <a href="#connect-to-the-server" id="connect-to-the-server"></a>

Using the key generated from the AWS console and using a SSH client like Putty, connect to your Ubuntu server. If you are logged in as root then create a user-level account with admin privileges instead, since logging in as the root user is risky.

Create a new user. Replace `yourusername` with a username of your choice. You will asked to create a strong password and provide some other optional information.

```
sudo adduser <yourusername>
```

Grant admin rights to the new user by adding it to the sudo group. This will allow the user to perform actions with superuser privileges by typing sudo before commands.

```
sudo usermod -aG sudo <yourusername>
```

Optional: If you used SSH keys to connect to your Ubuntu instance via the root user you will need to associate the new user with the root user’s SSH key data.

```
sudo rsync --archive --chown=<yourusername>:<yourusername> ~/.ssh /home/<yourusername>
```

Finally, log out of `root` and log in as `<yourusername>`

### Update the Server

Make sure the system is up to date with the latest software and security updates.&#x20;

```
sudo apt update && sudo apt upgrade
sudo apt dist-upgrade && sudo apt autoremove
sudo reboot
```

Enable automatic updates

```
sudo apt-get install unattended-upgrades
sudo dpkg-reconfigure -plow unattended-upgrades
```

**Optional**: Change the server's hostname with the following command:

```
sudo hostnamectl set-hostname <newNameHere>
```

## Secure the Server&#x20;

This guide will follow a list of settings in the [CoinCashew](https://www.coincashew.com/coins/overview-ada/guide-how-to-build-a-haskell-stakepool-node/how-to-harden-ubuntu-server) guide. This is not a comprehensive list and you should investigate other security steps specific to your own setup and situation.

### SSH Configuration

The following section will edit the contents of your `sshd_config` file, it is recommended to make a backup of this file before proceeding and to understand more about each of the settings you will edit you can view the sshd help file by typing:&#x20;

```
man sshd_config
```

Change the SSH **Port** in the `sshd_config` file

```
sudo nano /etc/ssh/sshd_config
```

Find the line `#Port 22` remove the `#` and replace the number with the port number of your choice

```
Port <YourSSHPortNumber>
```

{% hint style="info" %}
***At this point you may want to configure your firewall to allow the new port number.*** See the **Firewall Configuration** section below. Also remember to update any AWS Security Groups if you have those configured.
{% endhint %}

Locate the line for **ChallengeResponseAuthentication** and set it to ‘no’

```
ChallengeResponseAuthentication no
```

Locate the line for **PasswordAuthentication** and set it to ‘no’

```
PasswordAuthentication no
```

Locate the line for **PermitRootLogin** and set it to ‘no’

```
PermitRootLogin no
```

Locate the line for **PermitEmptyPasswords** and set it to ‘no’

```
PermitEmptyPasswords no
```

Locate the line for **PubkeyAuthentication** and set it to ‘yes’. This will change the configuration to only accept public keys.

```
PubkeyAuthentication yes
```

Change **ClientAliveInterval** to 300 and **ClientAliveCountMax** to 0

```
ClientAliveInterval 300
ClientAliveCountMax 0
```

Save and close the file, then test the SSH config.&#x20;

```
sudo sshd -t
```

If there are no errors then restart the SSH process

```
sudo systemctl restart sshd
```

Open another Putty terminal and test connecting on the new SSH port before closing the existing terminal.

#### Restricting access to specific users or IPs

{% hint style="warning" %}
**Optional Steps:** lockdown SSH to a specific user and/or from a specific IP. Only perform these step if you are confident your IP won’t change.
{% endhint %}

Open the SSHD config file again

```
sudo nano /etc/ssh/sshd_config
```

to restrict just to a specific user add the following to the bottom of the file

```
AllowUsers <SSH_User1> <SSH_User2> 
```

alternatively to restrict to a specific IP add the following line to the bottom of the file

```
AllowUsers <SSH_User>@<Public_IP>
```

Save and close the file and restart SSHD

```
sudo systemctl restart sshd
```

#### Restrict Access Using IP Tables

Alternatively, you can use IP tables to restrict access to the SSH port from a single IP or network. As root enter the follow at the command line

```
sudo iptables -A INPUT -p tcp --dport <YOUR_SSH_PORT> -s <Your_Public_IP> -j ACCEPT
sudo iptables -A INPUT -p tcp --dport <YOUR_SSH_PORT> -j DROP
```

### Create a new ED25519 encryption key and replace the default AWS RSA 2048-bit key

The default AWS SSH key is using **RSA SSH-2 2048-bit** encryption. It is recommended to create another key pair using **ED25519** encryption, follow the Putty user guide [here](https://docs.digitalocean.com/products/droplets/how-to/add-ssh-keys/create-with-putty/). Ensure you have set a strong password on your private key. Read up about ED25519 [here](https://medium.com/risan/upgrade-your-ssh-key-to-ed25519-c6e8d60d3c54).

Then once you’ve created the key copy the public key to the `~/.ssh/authorized_keys` file.&#x20;

```
sudo nano ~/.ssh/authorized_keys
```

save and close the file.

Test the new key works by opening a new SSH session via Putty and using the custom port you set earlier.

![](https://3418161995-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MeGGhDBZUaXZ9G17tnc%2F-MeGVdeBQVA1KuIIUdpu%2F-MeGXlqYzr4kRNx9DO_N%2Fimg_3.png?alt=media\&token=edab4c8c-6f35-44cc-b854-692b5c1c5dbc)

Once confirmed either backup and delete or comment (**`#`**) out the previous RSA key’s line in the `authorized_keys` file.

Ensure the `authorized_keys` file has the correct permissions by running the following command

```
chmod -R go= ~/.ssh
```

Check the permissions are set correctly

```
cd ~/.ssh
ls -l
```

![](https://3418161995-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MeGGhDBZUaXZ9G17tnc%2F-MeGNLmDuMfP-6VAQf_m%2F-MeGUol2cptvap7Rl-mc%2Fchmod.jpg?alt=media\&token=cffa0056-7fbb-4f17-866b-148087a4d23c)

### Disable the root account

Disable the ability to login with the `root`account using a password

```
sudo passwd -l root
```

### Firewall Configuration&#x20;

ufw is a a common linux based firewall package which we will install

Install the ufw package

```
sudo apt install ufw
```

Explicitly apply the defaults. Inbound traffic denied, outbound traffic allowed.

```
sudo ufw default deny incoming
sudo ufw default allow outgoing
```

Allow inbound traffic on `<YourSSHPortNumber>` as set in the **SSH Configuration** section above. SSH requires the TCP protocol.

```
sudo ufw allow <yourSSHportnumber>/tcp
```

{% hint style="warning" %}
**Optional:** You may also choose to lockdown access to your specific public IP. However be warned that if your public IP changes you may lose access.
{% endhint %}

```
sudo ufw allow proto tcp from <YourPublicIP> to any port <YourSSHPortNumber>
```

Deny inbound traffic on port 22/TCP.

{% hint style="danger" %}
***Only perform this step after confirming you have connected over SSH using \<YourSSHPortNumber>***
{% endhint %}

```
sudo ufw deny 22/tcp
```

Enable the firewall and check to verify the rules have been correctly configured.

```
sudo ufw enable
sudo ufw status numbered
```

### Two-factor authentication (optional but encouraged)&#x20;

Two-factor authentication gives you an extra layer of security in the event your SSH key was compromised. Although we are installing the **Google Authenticator** package you can also use an alternative 2fA app like **Authy**.

{% hint style="danger" %}

#### **Warning!!** *Ensure you complete all these steps without closing the Putty terminal. If you close the Putty terminal before you complete all the steps you may not be able to log back in again.*

{% endhint %}

Install the Google Authenticator package

```
sudo apt install libpam-google-authenticator -y
```

Run `google-authenticator` &#x20;

```
google-authenticator
```

enter ‘y’ to the first prompt

```
Do you want authentication tokens to be time-based (y/n): y
```

A QR Code will appear with a Secret Key, a number of scratch codes and a prompt for further options. Open your **Google Authenticator** or **Authy** app and enter in your secret key. Copy down your emergency scratch codes for safe-keeping.

The recommended settings for the subsequent prompts are:

```
Update the .google_authenticator file: yes
Disallow multiple uses: yes
Increase the original generation time limit: no
Enable rate-limiting: yes
```

Make a backup of the sshd configuration file

```
sudo cp /etc/pam.d/sshd /etc/pam.d/sshd.bak
```

Edit the /etc/pam.d/sshd file to make SSH use Google Auhthenticator

```
sudo nano /etc/pam.d/sshd
```

and add the following lines

```
@include common-password
auth required pam_google_authenticator.so nullok
auth required pam_permit.so
```

save and close the file.

Edit the SSH configuration file

```
sudo nano /etc/ssh/sshd_config
```

locate the `ChallengeResponseAuthentication` line and update to ‘yes’

```
KbdInteractiveAuthentication yes
```

locate the UsePAM line and update to ‘yes’

```
UsePAM yes
```

Save and close the file then restart the SSHD service

```
sudo systemctl restart sshd.service
```

{% hint style="warning" %}
2FA is not fully configured yet but open another Putty window to confirm all is working okay. You should not be prompted for any two-factor authentication yet. **DO NOT** close your existing terminal.
{% endhint %}

Make SSH aware of 2FA by opening the SSH configuration file

```
sudo nano /etc/ssh/sshd_config
```

and add the following line to the bottom of the file

```
AuthenticationMethods publickey,password publickey,keyboard-interactive
```

save and close the file.

Open the PAM sshd configuration file

```
sudo nano /etc/pam.d/sshd
```

and comment out the following line by adding a `#` character at the start

```
#@include common-auth
```

Save and close the file the restart SSH.

```
sudo systemctl restart sshd.service
```

Now open another Putty terminal session and you should be asked to enter the two-factor code.&#x20;

![](https://3418161995-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MeGGhDBZUaXZ9G17tnc%2F-MeGVdeBQVA1KuIIUdpu%2F-MeGawmmHZBnVCuAvVB3%2F2fa.jpg?alt=media\&token=7397e865-aeab-415d-b2fd-c6f4607cb8fd)

### Kernel Live Patching&#x20;

The Livepatch Service intends to address high and critical severity Linux kernel security vulnerabilities, as identified by Ubuntu Security Notices and the CVE tracker. Since there are limitations to the kernel livepatch technology, some Linux kernel code paths cannot be safely patched while running. There may be occasions when the traditional kernel upgrade and reboot might still be necessary.

You will need to create an account at <https://login.ubuntu.com/>. The free tier allows installation on up to three machines. You will be given a subscription token which can be found here <https://ubuntu.com/advantage>

Attach the token to your server

```
sudo ua attach <YOUR_TOKEN>
sudo ua status
```

### Secure Shared Memory

Shared memory can be used in an attack against a running service. Because of this, secure that portion of system memory. You can do this by modifying the **/etc/fstab** file.

Edit the fstab file

```
sudo nano /etc/fstab
```

Add the following line

```
tmpfs	/run/shm	tmpfs	ro,noexec,nosuid	0 0
```

Save and close the file, then reboot.

```
sudo reboot
```

### Install Fail2Ban&#x20;

Fail2ban is an intrusion-prevention system that monitors log files and searches for particular patterns that correspond to a failed login attempt. If a certain number of failed logins are detected from a specific IP address (within a specified amount of time), fail2ban blocks access from that IP address.

Install fail2ban

```
sudo apt-get install fail2ban -y
```

Edit the config file

```
sudo nano /etc/fail2ban/jail.local
```

and add the following to the file `ignoreip = <list of whitelisted IP address, your local daily laptop/pc>` . Also amend the port number to your own SSH port.

```
[sshd]
enabled = true
port = <22 or your random port number>
filter = sshd
logpath = /var/log/auth.log
maxretry = 3
# whitelisted IP addresses
ignoreip =  <list of whitelisted IP address, your local daily laptop/pc>
```

save and close the file.

Restart fail2ban

```
sudo systemctl restart fail2ban
```

That concludes the first step in preparing your Node Server. Head to the next page to install Node Exporter.

## Sources:

[Digital Ocean MFA Setup Link](https://www.digitalocean.com/community/tutorials/how-to-set-up-multi-factor-authentication-for-ssh-on-ubuntu-18-04)

[ED25519 Reference](https://medium.com/risan/upgrade-your-ssh-key-to-ed25519-c6e8d60d3c54)

[Coin Cashew’s Hardening Ubuntu Guide](https://www.coincashew.com/coins/overview-ada/guide-how-to-build-a-haskell-stakepool-node/how-to-harden-ubuntu-server)

[Digital Ocean Ubuntu Open SSH Hardening Tips](https://www.digitalocean.com/community/tutorials/how-to-harden-openssh-on-ubuntu-18-04)

[How to Harden your Ubuntu 18.04 Server](https://medium.com/@BaneBiddix/how-to-harden-your-ubuntu-18-04-server-ffc4b6658fe7)

[Linux hardening security tips](https://www.ubuntupit.com/best-linux-hardening-security-tips-a-comprehensive-checklist/)

[Kernal Live Patching - Florian Pieper](https://github.com/fpieper/fpstaking/blob/main/docs/validator_guide.md)


# Install Node Exporter

## Introduction

In this section we will install Node Exporter on the Radix Node Server. It will be used to scrape the Node Server and send CPU, RAM, Disk and  other system metrics plus Radix's network metrics to the monitoring web server. The steps here are based on Digital Ocean's guide which can be found [here](https://www.digitalocean.com/community/tutorials/how-to-install-prometheus-on-ubuntu-16-04).

### Install Node Exporter

We’ll begin by creating a **node\_exporter** user account.  Create the account with the `--no-create-home` and `--shell /bin/false` options so that these users can’t log into the server.

```bash
sudo useradd --no-create-home --shell /bin/false node_exporter
```

You can find the latest binaries along with their checksums on [Prometheus’ download page](https://prometheus.io/download/).

```
curl -LO https://github.com/prometheus/node_exporter/releases/download/v1.6.1/node_exporter-1.6.1.linux-amd64.tar.gz
```

&#x20;Use the `sha256sum` command to generate a checksum of the downloaded file:

```
sha256sum node_exporter-1.6.1.linux-amd64.tar.gz
```

Verify the downloaded file’s integrity by comparing its checksum with the one on the download page.

```
ecc41b3b4d53f7b9c16a370419a25a133e48c09dfc49499d63bcc0c5e0cf3d01  node_exporter-1.6.1.linux-amd64.tar.gz
```

![](https://3418161995-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MeGGhDBZUaXZ9G17tnc%2F-MfgdVrT5OlYGXIYGdsx%2F-Mfgr5urvIlE0j3jVKao%2Fimage.png?alt=media\&token=58417c58-f076-40bd-9fdc-02e0428a06e5)

Now, unpack the downloaded archive.

```bash
tar xvf node_exporter-1.6.1.linux-amd64.tar.gz
```

This will create a directory called `node_exporter-1.2.0.linux-amd64` containing a binary file named `node_exporter`, a license, and a notice.

Copy the binary to the `/usr/local/bin` directory and set the user and group ownership to the **node\_exporter** user that you created in Step 1.

```bash
sudo cp node_exporter-1.6.1.linux-amd64/node_exporter /usr/local/bin
sudo chown node_exporter:node_exporter /usr/local/bin/node_exporter
```

Lastly, remove the leftover files from your home directory as they are no longer needed.

```bash
rm -rf node_exporter-1.6.1.linux-amd64.tar.gz node_exporter-1.6.1.linux-amd64
```

Create a Systemd service file for Node Exporter.

```bash
sudo nano /etc/systemd/system/node_exporter.service
```

This service file tells your system to run Node Exporter as the **node\_exporter** user with the default set of collectors enabled.

Copy the following content into the service file:

{% code title="Node Exporter service file - /etc/systemd/system/node\_exporter.service" %}

```bash
[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter

[Install]
WantedBy=multi-user.target
```

{% endcode %}

Save the file and close your text editor.

Finally, reload `systemd` to use the newly created service.

```bash
sudo systemctl daemon-reload
```

&#x20;You can now run Node Exporter using the following command:

```bash
sudo systemctl start node_exporter
```

&#x20;Verify that Node Exporter’s running correctly with the `status` command.

```bash
sudo systemctl status node_exporter
```

&#x20;Like before, this output tells you Node Exporter’s status, main process identifier (PID), memory usage, and more.

If the service’s status isn’t `active`, follow the on-screen messages and re-trace the preceding steps to resolve the problem before continuing.

```bash
Output● node_exporter.service - Node Exporter
   Loaded: loaded (/etc/systemd/system/node_exporter.service; disabled; vendor preset: enabled)
   Active: active (running) since Fri 2017-07-21 11:44:46 UTC; 5s ago
 Main PID: 2161 (node_exporter)
    Tasks: 3
   Memory: 1.4M
      CPU: 11ms
   CGroup: /system.slice/node_exporter.service
```

Lastly, enable Node Exporter to start on boot.

```bash
sudo systemctl enable node_exporter
```

And that's it for preparing your Node Server. The next step is to install the RadixNode package on the server which can be found here

{% content-ref url="/pages/-Me\_yG5x8Uh9618nR-kO" %}
[Install the Radix Node](/install-and-configure-the-radix-validator-software/install-radix-node)
{% endcontent-ref %}


# Build and Configure the Monitoring Webserver

Steps to configure the standalone web server including installation of Prometheus, Grafana and Let's encrypt certificates.

{% hint style="info" %}
**Note:** These steps are identical to the Node Server configuration instructions with the exception of configuring Secure Shared Memory, Kernel Live Patching and 2FA. However you may consider these as necessary for your own requirements and wish to include them as well.
{% endhint %}

### Connect to the server <a href="#connect-to-the-server" id="connect-to-the-server"></a>

Using the key generated from the AWS console and using a SSH client like Putty, connect to your Ubuntu server. If you are logged in as root then create a user-level account with admin privileges instead, since logging in as the root user is risky.

Create a new user. Replace `yourusername` with a username of your choice. You will asked to create a strong password and provide some other optional information.

```
$ sudo adduser <yourusername>
```

Grant admin rights to the new user by adding it to the sudo group. This will allow the user to perform actions with superuser privileges by typing sudo before commands.

```
$ sudo usermod -aG sudo <yourusername>
```

Optional: If you used SSH keys to connect to your Ubuntu instance via the root user you will need to associate the new user with the root user’s SSH key data.

```
$ sudo rsync --archive --chown=<yourusername>:<yourusername> ~/.ssh /home/<yourusername>
```

Finally, log out of `root` and log in as `<yourusername>`

### Update the Server

Make sure the system is up to date with the latest software and security updates.&#x20;

```
$ sudo apt update && sudo apt upgrade
$ sudo apt dist-upgrade && sudo apt autoremove
$ sudo reboot
```

Enable automatic updates

```
$ sudo apt-get install unattended-upgrades
$ sudo dpkg-reconfigure -plow unattended-upgrades
```

Optional: Change the server's hostname with the following command:

```
$ sudo hostnamectl set-hostname newNameHere
```

## Secure the Server&#x20;

This guide will follow a list of settings in the [CoinCashew](https://www.coincashew.com/coins/overview-ada/guide-how-to-build-a-haskell-stakepool-node/how-to-harden-ubuntu-server) guide. This is not a comprehensive list and you should investigate other security steps specific to your own setup and situation.

### SSH Configuration

Change the Default SSH port from a port between 1024-49151. First check that the port is free

```
eg. sudo ss -tulpn | grep ':6673'

$ sudo ss -tulpn | grep ':<YourSSHPortNumber>'
```

A red text response indicates it's in use already.

If it's free then change the port in the `sshd_config` file

```
$ sudo nano /etc/ssh/sshd_config
```

Find the line `#Port 22` remove the `#` and replace the number with the port number of your choice

```
Port <YourSSHPortNumber>
```

{% hint style="info" %}
***At this point you may want to configure your firewall to allow the new port number.*** See the **Firewall Configuration** section below. Also remember to update any AWS Security Groups if you have those configured.
{% endhint %}

Locate the line for **ChallengeResponseAuthentication** and set it to ‘no’

```
ChallengeResponseAuthentication no
```

Locate the line for **PasswordAuthentication** and set it to ‘no’

```
PasswordAuthentication no
```

Locate the line for **PermitRootLogin** and set it to ‘no’

```
PermitRootLogin no
```

Locate the line for **PermitEmptyPasswords** and set it to ‘no’

```
PermitEmptyPasswords no
```

Locate the line for **PubkeyAuthentication** and set it to ‘yes’. This will change the configuration to only accept public keys.

```
PubkeyAuthentication yes
```

Save and close the file, then test the SSH config.&#x20;

```
$ sudo sshd -t
```

If there are no errors then restart the SSH process

```
$ sudo systemctl restart sshd
```

Log out and back in again using the new SSH port number.

{% hint style="warning" %}
**Optional Steps:** lockdown SSH to a specific user from a specific IP. Only perform this step if you are confident your IP won’t change.
{% endhint %}

Open the SSHD config file again

```
$ sudo nano /etc/ssh/sshd_config
```

and add the following line to the bottom of the file

```
AllowUsers <SSH_User>@<Public_IP>
```

Save and close the file and restart SSHD

```
$ sudo systemctl restart sshd
```

### Create a new ED25519 encryption key and replace the default AWS RSA 2048-bit key

The default AWS SSH key is using **RSA SSH-2 2048-bit** encryption. It is recommended to create another key pair using **ED25519** encryption, follow the Putty user guide [here](https://docs.digitalocean.com/products/droplets/how-to/add-ssh-keys/create-with-putty/). Ensure you have set a strong password on your private key. Then once you’ve created the key copy the public key to the `~/.ssh/authorized_keys` file. Test the new key works by opening a new SSH session via Putty and using the custom port you set earlier.

![](https://3418161995-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MeGGhDBZUaXZ9G17tnc%2F-MeGVdeBQVA1KuIIUdpu%2F-MeGXlqYzr4kRNx9DO_N%2Fimg_3.png?alt=media\&token=edab4c8c-6f35-44cc-b854-692b5c1c5dbc)

Once confirmed either backup and delete or comment (**`#`**) out the previous RSA key’s line in the `authorized_keys` file.

Read up about ED25519 [here](https://medium.com/risan/upgrade-your-ssh-key-to-ed25519-c6e8d60d3c54)

Ensure the `authorized_keys` file has the correct permissions by running the following command

```
chmod -R go= ~/.ssh
```

Check the permissions are set correctly

![](https://3418161995-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MeGGhDBZUaXZ9G17tnc%2F-MeGNLmDuMfP-6VAQf_m%2F-MeGUol2cptvap7Rl-mc%2Fchmod.jpg?alt=media\&token=cffa0056-7fbb-4f17-866b-148087a4d23c)

### Disable the root account

Disable the ability to login with the `root`account using a password

```
$ sudo passwd -l root
```

### Firewall Configuration&#x20;

ufw is a a common linux based firewall package which we will install

Install the ufw package

```
$ sudo apt install ufw
```

Explicitly apply the defaults. Inbound traffic denied, outbound traffic allowed.

```
$ sudo ufw default deny incoming
$ sudo ufw default allow outgoing
```

Allow inbound traffic on `<YourSSHPortNumber>` as set in the **SSH Configuration** section above. SSH requires the TCP protocol.

```
E.g. sudo ufw allow 6673/tcp
$ sudo ufw allow <yourSSHportnumber>/tcp
```

{% hint style="warning" %}
**Optional:** You may also choose to lockdown access to your specific public IP. However be warned that if your public IP changes you may lose access.
{% endhint %}

```
$ sudo ufw allow proto tcp from <YourPublicIP> to any port <YourSSHPortNumber>
```

Deny inbound traffic on port 22/TCP.

{% hint style="danger" %}
**Only perform this step after confirming you have connected over SSH using \<YourSSHPortNumber>**
{% endhint %}

```
$ sudo ufw deny 22/tcp
```

Enable the firewall and check to verify the rules have been correctly configured.

```
$ sudo ufw enable
$ sudo ufw status numbered
```

### Install Fail2Ban&#x20;

Fail2ban is an intrusion-prevention system that monitors log files and searches for particular patterns that correspond to a failed login attempt. If a certain number of failed logins are detected from a specific IP address (within a specified amount of time), fail2ban blocks access from that IP address.

Install fail2ban

```
$ sudo apt-get install fail2ban -y
```

Edit the config file

```
sudo nano /etc/fail2ban/jail.local
```

and add the following to the bottom of the file `ignoreip = <list of whitelisted IP address, your local daily laptop/pc>` . Also amend the port number to your own SSH port.

```
[sshd]
enabled = true
port = <22 or your random port number>
filter = sshd
logpath = /var/log/auth.log
maxretry = 3
# whitelisted IP addresses
ignoreip =  of whitelisted IP address, your local daily laptop/pc>
```

save and close the file.

Restart fail2ban

```
$ sudo systemctl restart fail2ban
```

You may also want to complete the additional steps to Secure Shared Memory and install Kernal Live Patching as we did with the Node Server.


# Monitoring and Alerting

## Introduction

This section will provide the steps to install Prometheus, Grafana, Nginx and Certbot for monitoring your node server plus provide a means to send alerts using Telegram and PagerDuty. The Prometheus steps are once again taken from Digital Ocean's guide [here,](https://www.digitalocean.com/community/tutorials/how-to-install-prometheus-on-ubuntu-16-04) Grafana steps [here](https://www.digitalocean.com/community/tutorials/how-to-install-and-secure-grafana-on-ubuntu-20-04), Blackbox Exporter steps here, Nginx steps [here](https://www.digitalocean.com/community/tutorials/how-to-install-nginx-on-ubuntu-20-04) and finally the Certbot/Let's encrypt guide [here](https://www.digitalocean.com/community/tutorials/how-to-secure-nginx-with-let-s-encrypt-on-ubuntu-18-04).

## NGINX

Before we install Prometheus we will need to install NGINX to serve the HTTP traffic.

```bash
sudo apt update
sudo apt install nginx
```

Before testing Nginx, the firewall software needs to be adjusted to allow access to the service. Nginx registers itself as a service with `ufw` upon installation, making it straightforward to allow Nginx access.

List the application configurations that `ufw` knows how to work with by typing:

```bash
sudo ufw app list
```

You should get a listing of the application profiles:

```bash
Available applications:
  Nginx Full
  Nginx HTTP
  Nginx HTTPS
  OpenSSH
```

As demonstrated by the output, there are three profiles available for Nginx:

* **Nginx Full**: This profile opens both port 80 (normal, unencrypted web traffic) and port 443 (TLS/SSL encrypted traffic)
* **Nginx HTTP**: This profile opens only port 80 (normal, unencrypted web traffic)
* **Nginx HTTPS**: This profile opens only port 443 (TLS/SSL encrypted traffic)

{% hint style="info" %}
It is recommended that you enable the most restrictive profile that will still allow the traffic you’ve configured. We will choose 'Full' to begin with. Once Testing is fully complete you may want to restrict this further by changing to 'HTTPS' and deleting 'Full'
{% endhint %}

&#x20;You can enable this by typing:

```bash
sudo ufw allow 'Nginx HTTPS'
```

You can verify the change by typing:

```bash
sudo ufw status
```

The output will indicate which traffic is allowed:

![](https://3418161995-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MeGGhDBZUaXZ9G17tnc%2F-MeUmdrw4sXnATYuSVMI%2F-MeV1_Pk6vGZTtEMEv6Q%2FNGINX.png?alt=media\&token=518a7b28-a3b3-433b-8b9e-c1b5241503b5)

## Prometheus

For security purposes, we’ll begin by creating the Prometheus user account, **`prometheus`**. We’ll use this account throughout the tutorial to isolate the ownership on Prometheus’ core files and directories.

Create these user, and use the `--no-create-home` and `--shell /bin/false` options so that these users can’t log into the server.

```
sudo useradd --no-create-home --shell /bin/false prometheus
```

&#x20;Before we download the Prometheus binaries, create the necessary directories for storing Prometheus’ files and data. Following standard Linux conventions, we’ll create a directory in `/etc` for Prometheus’ configuration files and a directory in `/var/lib` for its data.

```
sudo mkdir /etc/prometheus
sudo mkdir /var/lib/prometheus
```

&#x20;Now, set the user and group ownership on the new directories to the **`prometheus`** user.

```
sudo chown prometheus:prometheus /etc/prometheus
sudo chown prometheus:prometheus /var/lib/prometheus
```

&#x20;With our user and directories in place, we can now download Prometheus and then create the minimal configuration file to run Prometheus for the first time.

### Download Prometheus

First, download and unpack the current stable version of Prometheus into your home directory. You can find the latest binaries along with their checksums on the [Prometheus download page](https://prometheus.io/download/).

```
cd ~
curl -LO https://github.com/prometheus/prometheus/releases/download/v2.28.1/prometheus-2.28.1.linux-amd64.tar.gz
```

&#x20;Next, use the `sha256sum` command to generate a checksum of the downloaded file:

```
sha256sum prometheus-2.28.1.linux-amd64.tar.gz
```

Compare the output from this command with the checksum on the Prometheus download page to ensure that your file is both genuine and not corrupted.

![](https://3418161995-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MeGGhDBZUaXZ9G17tnc%2F-MeV8A7S73OP6JQxGXdK%2F-MeVFyecFogdL_n7dgVb%2Fprometheus.png?alt=media\&token=66b5b8d9-4e56-4096-83af-2dfa2e01853b)

Now, unpack the downloaded archive.

```bash
tar xvf prometheus-2.28.1.linux-amd64.tar.gz
```

This will create a directory called `prometheus-2.28.1.linux-amd64` containing two binary files (`prometheus` and `promtool`), `consoles` and `console_libraries` directories containing the web interface files, a license, a notice, and several example files.

Copy the two binaries to the `/usr/local/bin` directory.

```
sudo cp prometheus-2.28.1.linux-amd64/prometheus /usr/local/bin/
sudo cp prometheus-2.28.1.linux-amd64/promtool /usr/local/bin/
```

Set the user and group ownership on the binaries to the **`prometheus`** user created in Step 1.

```
sudo chown prometheus:prometheus /usr/local/bin/prometheus
sudo chown prometheus:prometheus /usr/local/bin/promtool
```

Copy the `consoles` and `console_libraries` directories to `/etc/prometheus`.

```
sudo cp -r prometheus-2.28.1.linux-amd64/consoles /etc/prometheus
sudo cp -r prometheus-2.28.1.linux-amd64/console_libraries /etc/prometheus
```

Set the user and group ownership on the directories to the **prometheus** user. Using the `-R` flag will ensure that ownership is set on the files inside the directory as well.

```
sudo chown -R prometheus:prometheus /etc/prometheus/consoles
sudo chown -R prometheus:prometheus /etc/prometheus/console_libraries
```

Lastly, remove the leftover files from your home directory as they are no longer needed.

```
rm -rf prometheus-2.28.1.linux-amd64.tar.gz prometheus-2.0.0.linux-amd64
```

Now that Prometheus is installed, we’ll create its configuration and service files in preparation of its first run.

### Configure Prometheus

In the `/etc/prometheus` directory, use `nano` or your favorite text editor to create a configuration file named `prometheus.yml`. For now, this file will contain just enough information to run Prometheus for the first time.

```bash
sudo nano /etc/prometheus/prometheus.yml
```

{% hint style="danger" %}
**Warning:** Prometheus’ configuration file uses the [YAML format](http://www.yaml.org/start.html), which strictly forbids tabs and requires two spaces for indentation. Prometheus will fail to start if the configuration file is incorrectly formatted.
{% endhint %}

In the `global` settings, define the default interval for scraping metrics. Note that Prometheus will apply these settings to every exporter unless an individual exporter’s own settings override the globals.

{% code title="Prometheus config file part 1 - /etc/prometheus/prometheus.yml global:" %}

```
 scrape_interval: 15s
```

{% endcode %}

This `scrape_interval` value tells Prometheus to collect metrics from its exporters every 15 seconds, which is long enough for most exporters.

Now, add Prometheus itself to the list of exporters to scrape from with the following `scrape_configs` directive:

Prometheus config file part 2 - /etc/prometheus/prometheus.yml

```
...
scrape_configs:
  - job_name: 'prometheus'
    scrape_interval: 5s
    static_configs:
      - targets: ['localhost:9090']
```

Prometheus uses the `job_name` to label exporters in queries and on graphs, so be sure to pick something descriptive here.

And, as Prometheus exports important data about itself that you can use for monitoring performance and debugging, we’ve overridden the global `scrape_interval` directive from 15 seconds to 5 seconds for more frequent updates.

Lastly, Prometheus uses the `static_configs` and `targets` directives to determine where exporters are running. Since this particular exporter is running on the same server as Prometheus itself, we can use `localhost` instead of an IP address along with the default port, `9090`.

Your configuration file should now look like this:

{% code title="Prometheus config file - /etc/prometheus/prometheus.yml" %}

```
global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    scrape_interval: 5s
    static_configs:
      - targets: ['localhost:9090']
```

{% endcode %}

Save the file and exit your text editor.

Now, set the user and group ownership on the configuration file to the **prometheus** user created in Step 1.

```
sudo chown prometheus:prometheus /etc/prometheus/prometheus.yml
```

With the configuration complete, we’re ready to test Prometheus by running it for the first time.

### Starting Prometheus

Start up Prometheus as the **`prometheus`** user, providing the path to both the configuration file and the data directory.

```
sudo -u prometheus /usr/local/bin/prometheus \
  --config.file /etc/prometheus/prometheus.yml \
  --storage.tsdb.path /var/lib/prometheus/ \
  --web.console.templates=/etc/prometheus/consoles \
  --web.console.libraries=/etc/prometheus/console_libraries
```

The output contains information about Prometheus’ loading progress, configuration file, and related services. It also confirms that Prometheus is listening on port `9090`.

```
Outputlevel=info ts=2017-11-17T18:37:27.474530094Z caller=main.go:215 msg="Starting Prometheus" version="(version=2.0.0, branch=HEAD, re
vision=0a74f98628a0463dddc90528220c94de5032d1a0)"
level=info ts=2017-11-17T18:37:27.474758404Z caller=main.go:216 build_context="(go=go1.9.2, user=root@615b82cb36b6, date=20171108-
07:11:59)"
level=info ts=2017-11-17T18:37:27.474883982Z caller=main.go:217 host_details="(Linux 4.4.0-98-generic #121-Ubuntu SMP Tue Oct 10 1
4:24:03 UTC 2017 x86_64 prometheus-update (none))"
level=info ts=2017-11-17T18:37:27.483661837Z caller=web.go:380 component=web msg="Start listening for connections" address=0.0.0.0
:9090
level=info ts=2017-11-17T18:37:27.489730138Z caller=main.go:314 msg="Starting TSDB"
level=info ts=2017-11-17T18:37:27.516050288Z caller=targetmanager.go:71 component="target manager" msg="Starting target manager...
"
level=info ts=2017-11-17T18:37:27.537629169Z caller=main.go:326 msg="TSDB started"
level=info ts=2017-11-17T18:37:27.537896721Z caller=main.go:394 msg="Loading configuration file" filename=/etc/prometheus/promethe
us.yml
level=info ts=2017-11-17T18:37:27.53890004Z caller=main.go:371 msg="Server is ready to receive requests."
```

If you get an error message, double-check that you’ve used YAML syntax in your configuration file and then follow the on-screen instructions to resolve the problem.

Now, halt Prometheus by pressing `CTRL+C`, and then open a new `systemd` service file.

```
sudo nano /etc/systemd/system/prometheus.service
```

The service file tells `systemd` to run Prometheus as the **prometheus** user, with the configuration file located in the `/etc/prometheus/prometheus.yml` directory and to store its data in the `/var/lib/prometheus` directory. (The details of `systemd` service files are beyond the scope of this tutorial, but you can learn more at [Understanding Systemd Units and Unit Files](https://www.digitalocean.com/community/tutorials/understanding-systemd-units-and-unit-files#where-are-systemd-unit-files-found).)

Copy the following content into the file:&#x20;

Prometheus service file - /etc/systemd/system/prometheus.service

```
[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
    --config.file /etc/prometheus/prometheus.yml \
    --storage.tsdb.path /var/lib/prometheus/ \
    --web.console.templates=/etc/prometheus/consoles \
    --web.console.libraries=/etc/prometheus/console_libraries

[Install]
WantedBy=multi-user.target
```

Finally, save the file and close your text editor.

To use the newly created service, reload `systemd`.

```
sudo systemctl daemon-reload
```

You can now start Prometheus using the following command:

```
sudo systemctl start prometheus
```

&#x20;To make sure Prometheus is running, check the service’s status.

```
sudo systemctl status prometheus
```

The output tells you Prometheus’ status, main process identifier (PID), memory use, and more.

If the service’s status isn’t `active`, follow the on-screen instructions and re-trace the preceding steps to resolve the problem before continuing the tutorial.

```
Output● prometheus.service - Prometheus
   Loaded: loaded (/etc/systemd/system/prometheus.service; disabled; vendor preset: enabled)
   Active: active (running) since Fri 2017-07-21 11:40:40 UTC; 3s ago
 Main PID: 2104 (prometheus)
    Tasks: 7
   Memory: 13.8M
      CPU: 470ms
   CGroup: /system.slice/prometheus.service
...
```

When you’re ready to move on, press `Q` to quit the `status` command.

Lastly, enable the service to start on boot.

```
sudo systemctl enable prometheus
```

Now that Prometheus is up and running, we can install an additional exporter to generate metrics about our server’s resources.

### Configure Prometheus to Scrape Node Exporter on the Node Server

Because Prometheus only scrapes exporters which are defined in the `scrape_configs` portion of its configuration file, we’ll need to add an entry for Node Exporter, just like we did for Prometheus itself.

{% hint style="info" %}
Before we do that however we need to open the firewall on the **Node** server to allow connections from the Monitoring server.
{% endhint %}

On the **Node** Server:

```
sudo ufw allow from <Your_Monitoring_Server_IP> to any port 9100
```

{% hint style="info" %}
Note: you may also need to open these ports within your AWS security groups
{% endhint %}

Open the configuration file.

```
sudo nano /etc/prometheus/prometheus.yml
```

&#x20;At the end of the `scrape_configs` block, add a new entry called `node_exporter`.

Prometheus config file part 1 - /etc/prometheus/prometheus.yml

```
...
  - job_name: 'node_exporter'
    scrape_interval: 5s
    static_configs:
      - targets: ['<YOUR_NODE_SERVER_IP>:9100']
```

&#x20;Because Node Exporter is running on the Node server , we need to add in `<YOUR_NODE_SERVER_IP>` with Node Exporter’s default port, `9100`.

Your whole configuration file should look like this:

Prometheus config file - /etc/prometheus/prometheus.yml

```
global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    scrape_interval: 5s
    static_configs:
      - targets: ['localhost:9090']
  - job_name: 'node_exporter'
    scrape_interval: 5s
    static_configs:
      - targets: ['<YOUR_NODE_SERVER_IP>:9100']
```

Save the file and exit your text editor when you’re ready to continue.

Finally, restart Prometheus to put the changes into effect.

```
sudo systemctl restart prometheus
```

Once again, verify that everything is running correctly with the `status` command.

```
sudo systemctl status prometheus
```

&#x20;If the service’s status isn’t set to `active`, follow the on screen instructions and re-trace your previous steps before moving on.

```
Output● prometheus.service - Prometheus
   Loaded: loaded (/etc/systemd/system/prometheus.service; disabled; vendor preset: enabled)
   Active: active (running) since Fri 2017-07-21 11:46:39 UTC; 6s ago
 Main PID: 2219 (prometheus)
    Tasks: 6
   Memory: 19.9M
      CPU: 433ms
   CGroup: /system.slice/prometheus.service
```

We now have Prometheus installed, configured, and running. As a final precaution before connecting to the web interface, we’ll enhance our installation’s security with basic HTTP authentication to ensure that unauthorized users can’t access our metrics.

### Securing Prometheus

Prometheus does not include built-in authentication or any other general purpose security mechanism. On the one hand, this means you’re getting a highly flexible system with fewer configuration restraints; on the other hand, it means it’s up to you to ensure that your metrics and overall setup are sufficiently secure.

For simplicity’s sake, we’ll use Nginx to add basic HTTP authentication to our installation, which both Prometheus and its preferred data visualization tool, Grafana, fully support.

Start by installing `apache2-utils`, which will give you access to the `htpasswd` utility for generating password files.

```
$ sudo apt-get update
$ sudo apt-get install apache2-utils
```

Now, create a password file by telling `htpasswd` where you want to store the file and which username `<username>` you’d like to use for authentication.

{% hint style="info" %}
**Note:** `htpasswd` will prompt you to enter and re-confirm the password you’d like to associate with this user. Also, make note of both the username and password you enter here, as you’ll need them to log into Prometheus in Step 9.
{% endhint %}

```
$ sudo htpasswd -c /etc/nginx/.htpasswd <username>
```

The result of this command is a newly-created file called `.htpasswd`, located in the `/etc/nginx` directory, containing the username and a hashed version of the password you entered.

Next, configure Nginx to use the newly-created passwords.

First, make a Prometheus-specific copy of the default Nginx configuration file so that you can revert back to the defaults later if you run into a problem.

```
sudo cp /etc/nginx/sites-available/default /etc/nginx/sites-available/prometheus
```

&#x20;Then, open the new configuration file.

```
sudo nano /etc/nginx/sites-available/prometheus
```

&#x20;Locate the `location /` block under the `server` block. It should look like:/etc/nginx/sites-available/default

```
...
    location / {
        try_files $uri $uri/ =404;
    }
...
```

&#x20;As we will be forwarding all traffic to Prometheus, replace the `try_files` directive with the following content:

{% code title="/etc/nginx/sites-available/prometheus" %}

```
...
    location / {
        auth_basic "Prometheus server authentication";
        auth_basic_user_file /etc/nginx/.htpasswd;
        proxy_pass http://localhost:9090;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_cache_bypass $http_upgrade;
    }
...
```

{% endcode %}

These settings ensure that users will have to authenticate at the start of each new session. Additionally, the reverse proxy will direct all requests handled by this block to Prometheus.

When you’re finished making changes, save the file and close your text editor.

Now, deactivate the default Nginx configuration file by removing the link to it in the `/etc/nginx/sites-enabled` directory, and activate the new configuration file by creating a link to it.

```
sudo rm /etc/nginx/sites-enabled/default
sudo ln -s /etc/nginx/sites-available/prometheus /etc/nginx/sites-enabled/
```

&#x20;Before restarting Nginx, check the configuration for errors using the following command:

```
sudo nginx -t
```

&#x20;The output should indicate that the `syntax is ok` and the `test is successful`. If you receive an error message, follow the on-screen instructions to fix the problem before proceeding to the next step.

Output of Nginx configuration tests:

```
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
```

Then, reload Nginx to incorporate all of the changes.

```
sudo systemctl reload nginx
```

Verify that Nginx is up and running.

```
sudo systemctl status nginx
```

If your output doesn’t indicate that the service’s status is `active`, follow the on-screen messages and re-trace the preceding steps to resolve the issue before continuing.

{% code title="Output" %}

```
● nginx.service - A high performance web server and a reverse proxy server
   Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: en
   Active: active (running) since Mon 2017-07-31 21:20:57 UTC; 12min ago
  Process: 4302 ExecReload=/usr/sbin/nginx -g daemon on; master_process on; -s r
 Main PID: 3053 (nginx)
    Tasks: 2
   Memory: 3.6M
      CPU: 56ms
   CGroup: /system.slice/nginx.service
```

{% endcode %}

At this point, we have a fully-functional and secured Prometheus server, so we can log into the web interface to begin looking at metrics.

### Testing Prometheus

Prometheus provides a basic web interface for monitoring the status of itself and its exporters, executing queries, and generating graphs. But, due to the interface’s simplicity, the Prometheus team [recommends](https://prometheus.io/docs/visualization/browser/) [installing and using Grafana](https://prometheus.io/docs/visualization/grafana/) for anything more complicated than testing and debugging.

In this tutorial, we’ll use the built-in web interface to ensure that Prometheus and Node Exporter are up and running before moving on to install Blackbox Exporter and Grafana.

To begin, point your web browser to `http://your_server_ip`.

In the HTTP authentication dialogue box, enter the username and password you chose earlier.

![Prometheus Authentication](https://assets.digitalocean.com/articles/install-prometheus-on-ubuntu-16-04/Prometheus-Authentication.png)

Once logged in, you’ll see the **Expression Browser**, where you can execute and visualize custom queries.

![Prometheus Dashboard Welcome](https://assets.digitalocean.com/articles/install-prometheus-on-ubuntu-16-04/Prometheus-Dashboard-Welcome.png)

Before executing any expressions, verify the status of both Prometheus and Node Explorer by clicking first on the **Status** menu at the top of the screen and then on the **Targets** menu option. As we have configured Prometheus to scrape both itself and Node Exporter, you should see both targets listed in the `UP` state.

![](https://3418161995-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MeGGhDBZUaXZ9G17tnc%2F-MeV3e51gOhErKXMm7Lx%2F-MeV7M3eIQak4Tnrof-G%2Fnode_exporter.png?alt=media\&token=77a2a128-4827-48bb-b59e-6529d081a188)

If either exporter is missing or displays an error message, check the service’s status with the following commands:

```
sudo systemctl status prometheus
```

```
sudo systemctl status node_exporter
```

The output for both services should report a status of `Active: active (running)`. If a service either isn’t active at all or is active but still not working correctly, follow the on-screen instructions and re-trace the previous steps before continuing.

## Installing Blackbox

The Blackbox exporter enables blackbox probing of endpoints over HTTP, HTTPS, DNS, TCP and ICMP. We can use it for checking the uptime status of both the Node and Monitoring servers.

### Create a Service User

For security purposes, we’ll create a **blackbox\_exporter** user account. We’ll use this account throughout the tutorial to run Blackbox Exporter and to isolate the ownership on appropriate core files and directories. This ensures Blackbox Exporter can't access and modify data it doesn't own.

Create these user with the `useradd` command using the `--no-create-home` and `--shell /bin/false` flags so that these users can’t log into the server:

```
sudo useradd --no-create-home --shell /bin/false blackbox_exporter
```

With the users in place, let’s download and configure Blackbox Exporter.

### Installing Blackbox Exporter <a href="#step-2-installing-blackbox-exporter" id="step-2-installing-blackbox-exporter"></a>

First, download the latest stable version of Blackbox Exporter to your home directory. You can find the latest binaries along with their checksums on the [Prometheus Download page](https://prometheus.io/download/).

```
cd ~
curl -LO https://github.com/prometheus/blackbox_exporter/releases/download/v0.12.0/blackbox_exporter-0.19.0.linux-amd64.tar.gz
```

Before unpacking the archive, verify the file’s checksums using the following `sha256sum` command:

```
sha256sum blackbox_exporter-0.19.0.linux-amd64.tar.gz
```

Compare the output from this command with the checksum on the [Prometheus download page](https://prometheus.io/download/) to ensure that your file is both genuine and not corrupted:

![](https://3418161995-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MeGGhDBZUaXZ9G17tnc%2F-MeV8A7S73OP6JQxGXdK%2F-MeVFauxnSrN2d-0gFvp%2Fblackbox.png?alt=media\&token=1cb3e737-eafa-4357-bc34-4034ecea95e4)

If the checksums don’t match, remove the downloaded file and repeat the preceding steps to re-download the file.

When you’re sure the checksums match, unpack the archive:

```
tar xvf blackbox_exporter-0.19.0.linux-amd64.tar.gz
```

This creates a directory called `blackbox_exporter-0.19.0.linux-amd64`, containing the `blackbox_exporter` binary file, a license, and example files.

Copy the binary file to the `/usr/local/bin` directory.

```
sudo mv ./blackbox_exporter-0.19.0.linux-amd64/blackbox_exporter /usr/local/bin
```

Set the user and group ownership on the binary to the **blackbox\_exporter** user, ensuring non-root users can’t modify or replace the file:

```
sudo chown blackbox_exporter:blackbox_exporter /usr/local/bin/blackbox_exporter
```

Lastly, we’ll remove the archive and unpacked directory, as they’re no longer needed.

```
rm -rf ~/blackbox_exporter-0.19.0.linux-amd64.tar.gz ~/blackbox_exporter-0.19.0.linux-amd64
```

Next, let’s configure Blackbox Exporter to probe endpoints over the HTTP protocol and then run it.

### Configuring and Running Blackbox Exporter <a href="#step-3-configuring-and-running-blackbox-exporter" id="step-3-configuring-and-running-blackbox-exporter"></a>

Let’s create a configuration file defining how Blackbox Exporter should check endpoints. We’ll also create a systemd unit file so we can manage Blackbox’s service using `systemd`.

We’ll specify the list of endpoints to probe in the Prometheus configuration in the next step.

First, create the directory for Blackbox Exporter’s configuration. Per Linux conventions, configuration files go in the `/etc` directory, so we’ll use this directory to hold the Blackbox Exporter configuration file as well:

```
sudo mkdir /etc/blackbox_exporter
```

Then set the ownership of this directory to the **blackbox\_exporter** user you created in Step 1:

```
sudo chown blackbox_exporter:blackbox_exporter /etc/blackbox_exporter
```

In the newly-created directory, create the `blackbox.yml` file which will hold the Blackbox Exporter configuration settings:

```
sudo nano /etc/blackbox_exporter/blackbox.yml
```

We’ll configure Blackbox Exporter to use the default `http` prober to probe endpoints. *Probers* define how Blackbox Exporter checks if an endpoint is running. The `http` prober checks endpoints by sending a HTTP request to the endpoint and testing its response code. You can select which HTTP method to use for probing, as well as which status codes to accept as successful responses. Other popular probers include the `tcp` prober for probing via the TCP protocol, the `icmp` prober for probing via the ICMP protocol and the `dns` prober for checking DNS entries.

For this tutorial, we’ll use the `http` prober to probe the endpoint running on port `8080` over the HTTP `GET` method. By default, the prober assumes that valid status codes in the `2xx` range are valid, so we don’t need to provide a list of valid status codes.

We’ll configure a timeout of **5** seconds, which means Blackbox Exporter will wait 5 seconds for the response before reporting a failure. Depending on your application type, choose any value that matches your needs.

{% hint style="info" %}
**Note:** Blackbox Exporter’s configuration file uses the [YAML format](http://www.yaml.org/start.html), which forbids using tabs and strictly requires using two spaces for indentation. If the configuration file is formatted incorrectly, Blackbox Exporter will fail to start up.
{% endhint %}

Add the following configuration to the file:

/etc/blackbox\_exporter/blackbox.yml

```
modules:
  http_2xx:
    prober: http
    timeout: 5s
    http:
      valid_status_codes: []
      method: GET
```

You can find more information about the configuration options in the [the Blackbox Exporter’s documentation](https://github.com/prometheus/blackbox_exporter/blob/master/CONFIGURATION.md).

Save the file and exit your text editor.

Before you create the service file, set the user and group ownership on the configuration file to the **blackbox\_exporter** user created in Step 1.

```
sudo chown blackbox_exporter:blackbox_exporter /etc/blackbox_exporter/blackbox.yml
```

Now create the service file so you can manage Blackbox Exporter using `systemd`:

```
sudo nano /etc/systemd/system/blackbox_exporter.service
```

Add the following content to the file:

/etc/systemd/system/blackbox\_exporter.service

```
[Unit]
Description=Blackbox Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=blackbox_exporter
Group=blackbox_exporter
Type=simple
ExecStart=/usr/local/bin/blackbox_exporter --config.file /etc/blackbox_exporter/blackbox.yml

[Install]
WantedBy=multi-user.target
```

This service file tells `systemd` to run Blackbox Exporter as the **blackbox\_exporter** user with the configuration file located at `/etc/blackbox_exporter/blackbox.yml`. The details of `systemd` service files are beyond the scope of this tutorial, but if you’d like to learn more see the [Understanding Systemd Units and Unit Files](https://www.digitalocean.com/community/tutorials/understanding-systemd-units-and-unit-files#where-are-systemd-unit-files-found) tutorial.

Save the file and exit your text editor.

Finally, reload `systemd` to use your newly-created service file:

```
sudo systemctl daemon-reload
```

Now start Blackbox Exporter:

```
sudo systemctl start blackbox_exporter
```

Make sure it started successfully by checking the service’s status:

```
sudo systemctl status blackbox_exporter
```

The output contains information about Blackbox Exporter’s process, including the main process identifier (PID), memory use, logs and more.

```
Output● blackbox_exporter.service - Blackbox Exporter
   Loaded: loaded (/etc/systemd/system/blackbox_exporter.service; disabled; vendor preset: enabled)
   Active: active (running) since Thu 2018-04-05 17:48:58 UTC; 5s ago
 Main PID: 5869 (blackbox_export)
    Tasks: 4
   Memory: 968.0K
      CPU: 9ms
   CGroup: /system.slice/blackbox_exporter.service
           └─5869 /usr/local/bin/blackbox_exporter --config.file /etc/blackbox_exporter/blackbox.yml
```

If the service’s status isn’t `active (running)`, follow the on-screen logs and retrace the preceding steps to resolve the problem before continuing the tutorial.

Lastly, enable the service to make sure Blackbox Exporter will start when the server restarts:

```
sudo systemctl enable blackbox_exporter
```

Now that Blackbox Exporter is fully configured and running, we can configure Prometheus to collect metrics about probing requests to our endpoint, so we can create alerts based on those metrics and set up notifications for alerts using Alertmanager.

### Configuring Prometheus To Scrape Blackbox Exporter <a href="#step-4-configuring-prometheus-to-scrape-blackbox-exporter" id="step-4-configuring-prometheus-to-scrape-blackbox-exporter"></a>

As mentioned in Step 3, the list of endpoints to be probed is located in the Prometheus configuration file as part of the Blackbox Exporter’s `targets` directive. In this step you’ll configure Prometheus to use Blackbox Exporter to scrape the Nginx web server running on port `80` that you configured in the prerequisite tutorials.

Open the Prometheus configuration file in your editor:

```
sudo nano /etc/prometheus/prometheus.yml
```

At this point, it should look like the following:

/etc/prometheus/prometheus.yml

```
global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    scrape_interval: 5s
    static_configs:
      - targets: ['localhost:9090']
  - job_name: 'node_exporter'
    scrape_interval: 5s
    static_configs:
      - targets: ['<YOUR_NODE_SERVER_IP>:9100']
```

At the end of the `scrape_configs` directive, add the following entry, which will tell Prometheus to probe the endpoint running on the local port `80` using the Blackbox Exporter’s module `http_2xx`, configured in Step 3.

/etc/prometheus/prometheus.yml

```
...
  - job_name: 'blackbox'
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets:
        - http://localhost:80
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: localhost:9115
```

By default, Blackbox Exporter runs on port `9115` with metrics available on the `/probe` endpoint.

The `scrape_configs` configuration for Blackbox Exporter differs from the configuration for other exporters. The most notable difference is the `targets` directive, which lists the endpoints being probed instead of the exporter’s address. The exporter’s address is specified using the appropriate set of `__address__` labels.

You’ll find a detailed explanation of the `relabel` directives in the [Prometheus documentation](https://prometheus.io/docs/introduction/overview/).

Your Prometheus configuration file will now look like this:Prometheus config file - /etc/prometheus/prometheus.yml

```
global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    scrape_interval: 5s
    static_configs:
      - targets: ['localhost:9090']
  - job_name: 'node_exporter'
    scrape_interval: 5s
    static_configs:
      - targets: ['<YOUR_NODE_SERVER_IP>:9100']
  - job_name: 'blackbox'
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets:
        - http://localhost:80
        - http://<YOUR_NODE_SERVER_IP>:9100
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: localhost:9115
```

Save the file and close your text editor.

Restart Prometheus to put the changes into effect:

```
sudo systemctl restart prometheus
```

Make sure it’s running as expected by checking the Prometheus service status:

```
sudo systemctl status prometheus
```

If the service’s status isn’t `active (running)`, follow the on-screen logs and retrace the preceding steps to resolve the problem before continuing the tutorial.

At this point, you’ve configured Prometheus to scrape metrics from Blackbox Exporter.&#x20;

## Installing Grafana

[Grafana](https://grafana.com/grafana) is an open-source data visualization and monitoring tool that we will integrate with [Prometheus](https://prometheus.io/) to provide a graphical representation of the data being pulled from the Node Server.&#x20;

It will require the following:

* A registered Domain name from a Domain Registrar.&#x20;
* An **A** record with `your_domain` pointing to your server’s public IP address.
* An **A** record with `www.your_domain` pointing to your server’s public IP address.
* Nginx installed and configured
* Installation of a Let's Encrypt SSL certificate with Certbot&#x20;
* Ensure port 443 is open for the Monitoring server within your AWS security groups
* You may also need to open port 80 within your AWS Security groups temporarily until SSL has been configured and port 443 is available

### Complete Nginx Configuration

The first step is to install Nginx which we have partially completed prior to installing Prometheus. We will pick up where we left off. These steps are taken from Digital Ocean's Nginx guide [here](https://www.digitalocean.com/community/tutorials/how-to-install-nginx-on-ubuntu-20-04).

We can check with the `systemd` init system to make sure the service is running by typing:

```
systemctl status nginx
```

You should receive the following showing the service is `active`

```
● nginx.service - A high performance web server and a reverse proxy server
   Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
   Active: active (running) since Fri 2020-04-20 16:08:19 UTC; 3 days ago
     Docs: man:nginx(8)
 Main PID: 2369 (nginx)
    Tasks: 2 (limit: 1153)
   Memory: 3.5M
   CGroup: /system.slice/nginx.service
           ├─2369 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
           └─2380 nginx: worker process
```

### Setting Up Server Blocks (Recommended) <a href="#step-5-setting-up-server-blocks-recommended" id="step-5-setting-up-server-blocks-recommended"></a>

When using the Nginx web server, *server blocks* (similar to virtual hosts in Apache) can be used to encapsulate configuration details and host more than one domain from a single server. We will set up a domain called **your\_domain**, but you should **replace this with your own domain name**.&#x20;

Nginx on Ubuntu 20.04 has one server block enabled by default that is configured to serve documents out of a directory at `/var/www/html`. While this works well for a single site, it can become unwieldy if you are hosting multiple sites. Instead of modifying `/var/www/html`, let’s create a directory structure within `/var/www` for our **your\_domain** site, leaving `/var/www/html` in place as the default directory to be served if a client request doesn’t match any other sites.

Create the directory for **your\_domain** as follows, using the `-p` flag to create any necessary parent directories:

```
sudo mkdir -p /var/www/your_domain/html
```

Next, assign ownership of the directory with the `$USER` environment variable:

```
sudo chown -R $USER:$USER /var/www/your_domain/html
```

The permissions of your web roots should be correct if you haven’t modified your `umask` value, which sets default file permissions. To ensure that your permissions are correct and allow the owner to read, write, and execute the files while granting only read and execute permissions to groups and others, you can input the following command:

```
sudo chmod -R 755 /var/www/your_domain
```

Next, create a sample `index.html` page using `nano` or your favorite editor:

```
nano /var/www/your_domain/html/index.html
```

Inside, add the following sample HTML:/var/www/your\_domain/html/index.html

```
<html>
    <head>
        <title>Welcome to your_domain!</title>
    </head>
    <body>
        <h1>Success!  The your_domain server block is working!</h1>
    </body>
</html>
```

Save and close the file by typing `CTRL` and `X` then `Y` and `ENTER` when you are finished.

In order for Nginx to serve this content, it’s necessary to create a server block with the correct directives. Instead of modifying the default configuration file directly, let’s make a new one at `/etc/nginx/sites-available/your_domain`:

```
sudo nano /etc/nginx/sites-available/your_domain
```

Paste in the following configuration block, which is similar to the default, but updated for our new directory and domain name:/etc/nginx/sites-available/your\_domain

```
server {
        listen 80;
        listen [::]:80;

        root /var/www/your_domain/html;
        index index.html index.htm index.nginx-debian.html;

        server_name your_domain www.your_domain;

        location / {
                try_files $uri $uri/ =404;
        }
}
```

&#x20;Notice that we’ve updated the `root` configuration to our new directory, and the `server_name` to our domain name.

Next, let’s enable the file by creating a link from it to the `sites-enabled` directory, which Nginx reads from during startup:

```
sudo ln -s /etc/nginx/sites-available/your_domain /etc/nginx/sites-enabled/
```

&#x20;Two server blocks are now enabled and configured to respond to requests based on their `listen` and `server_name` directives (you can read more about how Nginx processes these directives [here](https://www.digitalocean.com/community/tutorials/understanding-nginx-server-and-location-block-selection-algorithms)):

* `your_domain`: Will respond to requests for `your_domain` and `www.your_domain`.
* `default`: Will respond to any requests on port 80 that do not match the other two blocks.

To avoid a possible hash bucket memory problem that can arise from adding additional server names, it is necessary to adjust a single value in the `/etc/nginx/nginx.conf` file. Open the file:

```
sudo nano /etc/nginx/nginx.conf
```

Find the `server_names_hash_bucket_size` directive and remove the `#` symbol to uncomment the line. If you are using nano, you can quickly search for words in the file by pressing `CTRL` and `w`./etc/nginx/nginx.conf

```
...
http {
    ...
    server_names_hash_bucket_size 64;
    ...
}
...
```

&#x20;Save and close the file when you are finished.

Next, test to make sure that there are no syntax errors in any of your Nginx files:

```
sudo nginx -t
```

&#x20;If there aren’t any problems, restart Nginx to enable your changes:

```
sudo systemctl restart nginx
```

&#x20;Nginx should now be serving your domain name. You can test this by navigating to `http://your_domain`, where you should see something like this:

![Nginx first server block](https://assets.digitalocean.com/articles/nginx_server_block_1404/first_block.png)

## Install Let's Encrypt SSL Certificates with Certbot

Let's Encrypt is a free SSL service that can be installed on Linux hosts as an easy way to secure websites. The installation steps are taken from Certbot's guide [here](https://certbot.eff.org/lets-encrypt/ubuntufocal-nginx).

Ensure your version of snapd is up-to-date

```
sudo snap install core; sudo snap refresh core
```

Install certbot

```
sudo snap install --classic certbot
```

![](https://3418161995-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MeGGhDBZUaXZ9G17tnc%2F-MeW6CvzfyAYbI8rJcVR%2F-MeW9ylFR-i1zr9USnrv%2Fimage.png?alt=media\&token=0c7c5615-2a1a-4f86-a2c4-7f31926a8f4b)

Execute the following instruction on the command line on the machine to ensure that the certbot command can be run.

```
sudo ln -s /snap/bin/certbot /usr/bin/certbot
```

Run this command to get a certificate and have Certbot edit your Nginx configuration automatically to serve it, turning on HTTPS access in a single step.

```
sudo certbot --nginx
```

select both domains by entering `1,2`

![](https://3418161995-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MeGGhDBZUaXZ9G17tnc%2F-MfdvfIlN2jTMVziVXWG%2F-Mfdx1uKHZ9cK2T3YeNU%2Fimage.png?alt=media\&token=ad50f1b3-189f-4231-b3f9-64b6c3cae13c)

Choose to re-direct HTTP to HTTPS

![](https://3418161995-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MeGGhDBZUaXZ9G17tnc%2F-MfdvfIlN2jTMVziVXWG%2F-MfdxKI4Bly_fdFC0Lmg%2Fimage.png?alt=media\&token=f5a71664-b19c-491b-ab6a-8421e7c8d9ec)

Congrats! your certificate has been installed.

![](https://3418161995-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MeGGhDBZUaXZ9G17tnc%2F-MfdvfIlN2jTMVziVXWG%2F-MfdwnnUllBUj2Rq2jZK%2Fimage.png?alt=media\&token=e128408a-f4cb-46af-b211-4c00c982ebc2)

Optional: Test the certificate's strength using Qualys.

![](https://3418161995-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MeGGhDBZUaXZ9G17tnc%2F-MfdvfIlN2jTMVziVXWG%2F-MfdwS5Zdl5NwJgpjVq7%2Fimage.png?alt=media\&token=4cad88ec-f62c-4e85-9b21-804b552aeffb)

{% hint style="info" %}
Note: The certificate will expire in three months and would normally auto-renew. You can test the auto-renewal process will work by entering the following command
{% endhint %}

You can test automatic renewal for your certificates by running this command:

```
sudo certbot renew --dry-run
```

### Configuring Grafana

Download the Grafana [GPG key](https://www.digitalocean.com/community/tutorials/how-to-use-gpg-to-encrypt-and-sign-messages) with [`wget`](https://www.gnu.org/software/wget/), then [pipe the output](https://www.digitalocean.com/community/tutorials/an-introduction-to-linux-i-o-redirection#pipes) to `apt-key`. This will add the key to your APT installation’s list of trusted keys, which will allow you to download and verify the GPG-signed Grafana package:

```
wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -
```

In this command, the option `-q` turns off the status update message for `wget`, and `-O` outputs the file that you downloaded to the terminal. These two options ensure that only the contents of the downloaded file are pipelined to `apt-key`.

Next, add the Grafana repository to your APT sources:

```
sudo add-apt-repository "deb https://packages.grafana.com/oss/deb stable main"
```

&#x20;Refresh your APT cache to update your package lists:

```
sudo apt update
```

&#x20;You can now proceed with the installation:

```
sudo apt install grafana
```

Once Grafana is installed, use `systemctl` to start the Grafana server:

```
sudo systemctl start grafana-server
```

&#x20;Next, verify that Grafana is running by checking the service’s status:

```
sudo systemctl status grafana-server
```

&#x20;You will receive output similar to this:

```
Output● grafana-server.service - Grafana instance
     Loaded: loaded (/lib/systemd/system/grafana-server.service; disabled; vendor preset: enabled)
   Active: active (running) since Thu 2020-05-21 08:08:10 UTC; 4s ago
     Docs: http://docs.grafana.org
 Main PID: 15982 (grafana-server)
    Tasks: 7 (limit: 1137)
...
```

This output contains information about Grafana’s process, including its status, Main Process Identifier (PID), and more. `active (running)` shows that the process is running correctly.

Lastly, enable the service to automatically start Grafana on boot:

```
sudo systemctl enable grafana-server
```

&#x20;You will receive the following output:

```
OutputSynchronizing state of grafana-server.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable grafana-server
Created symlink /etc/systemd/system/multi-user.target.wants/grafana-server.service → /usr/lib/systemd/system/grafana-server.service.
```

This confirms that `systemd` has created the necessary symbolic links to autostart Grafana.

Grafana is now installed and ready for use. Next, you wil secure your connection to Grafana with a reverse proxy and SSL certificate.

### Setting Up the Reverse Proxy <a href="#step-2-setting-up-the-reverse-proxy" id="step-2-setting-up-the-reverse-proxy"></a>

Using an SSL certificate will ensure that your data is secure by encrypting the connection to and from Grafana. But, to make use of this connection, you’ll first need to reconfigure Nginx as a reverse proxy for Grafana.

Open the Nginx configuration file you created when you set up the Nginx server block with Let’s Encrypt in the [Prerequisites](https://www.digitalocean.com/community/tutorials/how-to-install-and-secure-grafana-on-ubuntu-20-04#prerequisites). You can use any text editor, but for this tutorial we’ll use `nano`:

```
sudo nano /etc/nginx/sites-available/your_domain
```

&#x20;Locate the following block:/etc/nginx/sites-available/your\_domain

```
...
    location / {
        try_files $uri $uri/ =404;
    }
...
```

&#x20;Because you already configured Nginx to communicate over SSL and because all web traffic to your server already passes through Nginx, you just need to tell Nginx to forward all requests to Grafana, which runs on port `3000` by default.

Delete the existing `try_files` line in this `location block` and replace it with the following `proxy_pass` option:/etc/nginx/sites-available/your\_domain

```
...
    location / {
        proxy_pass http://localhost:3000;
    }
...
```

&#x20;This will map the proxy to the appropriate port. Once you’re done, save and close the file by pressing `CTRL+X`, `Y`, and then `ENTER` if you’re using `nano`.

Now, test the new settings to make sure everything is configured correctly:

```
sudo nginx -t
```

You will receive the following output:

```
Outputnginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
```

Finally, activate the changes by reloading Nginx:

```
sudo systemctl reload nginx
```

&#x20;You can now access the default Grafana login screen by pointing your web browser to `https://your_domain`. If you’re unable to reach Grafana, verify that your firewall is set to allow traffic on port `443` and then re-trace the previous instructions.

With the connection to Grafana encrypted, you can now implement additional security measures, starting with changing Grafana’s default administrative credentials.

### Updating Credentials <a href="#step-3-updating-credentials" id="step-3-updating-credentials"></a>

Because every Grafana installation uses the same administrative credentials by default, it is a best practice to change your login information as soon as possible. In this step, you’ll update the credentials to improve security.

Start by navigating to `https://your_domain` from your web browser. This will bring up the default login screen where you’ll see the Grafana logo, a form asking you to enter an **Email or username** and **Password**, a **Log in** button, and a **Forgot your password?** link.

![Grafana Login](https://assets.digitalocean.com/articles/67242/grafana_login.png)

Enter `admin` into both the **Email or username** and **Password** fields and then click on the **Log in** button.

On the next screen, you’ll be asked to make your account more secure by changing the default password:

![Change Password](https://assets.digitalocean.com/articles/67242/change_password.png)

Enter the password you’d like to start using into the **New password** and **Confirm new password** fields.

From here, you can click **Submit** to save the new information or press **Skip** to skip this step. If you skip, you will be prompted to change the password next time you log in.

In order to increase the security of your Grafana setup, click **Submit**. You’ll go to the **Welcome to Grafana** dashboard:

![Home Dashboard](https://assets.digitalocean.com/articles/67242/home_dashboard.png)

You’ve now secured your account by changing the default credentials. Next, you will make changes to your Grafana configuration so that nobody can create a new Grafana account without your permission.

### Disabling Grafana Registrations and Anonymous Access <a href="#step-4-disabling-grafana-registrations-and-anonymous-access" id="step-4-disabling-grafana-registrations-and-anonymous-access"></a>

Grafana provides options that allow visitors to create user accounts for themselves and preview dashboards without registering. When Grafana isn’t accessible via the internet or when it’s working with publicly available data like service statuses, you may want to allow these features. However, when using Grafana online to work with sensitive data, anonymous access could be a security problem. To fix this problem, make some changes to your Grafana configuration.

Start by opening Grafana’s main configuration file for editing:

```
sudo nano /etc/grafana/grafana.ini
```

&#x20;Locate the following `allow_sign_up` directive under the `[users]` heading:/etc/grafana/grafana.ini

```
...
[users]
# disable user signup / registration
;allow_sign_up = true
...
```

&#x20;Enabling this directive with `true` adds a **Sign Up** button to the login screen, allowing users to register themselves and access Grafana.

Disabling this directive with `false` removes the **Sign Up** button and strengthens Grafana’s security and privacy.

Uncomment this directive by removing the `;` at the beginning of the line and then setting the option to `false`:/etc/grafana/grafana.ini

```
...
[users]
# disable user signup / registration
allow_sign_up = false
...
```

&#x20;Next, locate the following `enabled` directive under the `[auth.anonymous]` heading:/etc/grafana/grafana.ini

```
...
[auth.anonymous]
# enable anonymous access
;enabled = false
...
```

&#x20;Setting `enabled` to `true` gives non-registered users access to your dashboards; setting this option to `false` limits dashboard access to registered users only.

Uncomment this directive by removing the `;` at the beginning of the line and then setting the option to `false`./etc/grafana/grafana.ini

```
...
[auth.anonymous]
# enable anonymous access
enabled = false
...
```

&#x20;Save the file and exit your text editor.

To activate the changes, restart Grafana:

```
sudo systemctl restart grafana-server
```

Verify that everything is working by checking Grafana’s service status:

```
sudo systemctl status grafana-server
```

Like before, the output will report that Grafana is `active (running)`.

Now, point your web browser to `https://your_domain`. To return to the **Sign Up** screen, bring your cursor to your avatar in the lower left of the screen and click on the **Sign out** option that appears.

Once you have signed out, verify that there is no **Sign Up** button and that you can’t sign in without entering login credentials.

At this point, Grafana is fully configured and ready for use.

### Add a Dashboard

We will add two dashboards&#x20;

1. The default Grafana '[**Node Exporter Full**](https://grafana.com/grafana/dashboards/1860)' dashboard and
2. The Radix team provided '**Radix Node**' dashboard (optional)

Note: There are more dashboards available from the Grafana [website](https://grafana.com/grafana/dashboards)

#### Configuring a Data Source

Before we import a dashboard we need to connect to our Prometheus data source. From the Grafana homepage, click the cog icon, then 'Data Sources' and select 'Prometheus'

![Add the Prometheus data source](https://3418161995-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MeGGhDBZUaXZ9G17tnc%2F-MeWpBfO-agOKnHhv36E%2F-MeZrV0gQveVzKmCygUK%2Fimage.png?alt=media\&token=d365f1f4-2ac5-48d1-87dd-331c15d94d2b)

Leave the HTTP URL as the default <http://localhost:9090>

![Choose the defaults](https://3418161995-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MeGGhDBZUaXZ9G17tnc%2F-MeWpBfO-agOKnHhv36E%2F-MeZryFlurEcke0g6E8O%2Fimage.png?alt=media\&token=84b7c73f-6f71-4418-a332-e100bbdd7843)

Scroll to the bottom and click 'Save & Test'

![Test the data source](https://3418161995-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MeGGhDBZUaXZ9G17tnc%2F-MeWpBfO-agOKnHhv36E%2F-MeZsYYosmU83cszaFXu%2Fimage.png?alt=media\&token=07a0554d-23ad-4fee-ba52-b2203a603b71)

#### Import Node Exporter Full Dashboard

The next step is to setup your Grafana dashboards

Each dashboard on the Grafana website has an id, the '[Node Exporter Full](https://grafana.com/grafana/dashboards/1860)' dashboard has an id of `1860`. From the main Grafana window, click on the '+' icon, followed by 'Import' and then enter the id `1860` and 'Load'.

![Import Node Exporter Dashboard](https://3418161995-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MeGGhDBZUaXZ9G17tnc%2F-MeWTc3Q_o7cIHNKwNxd%2F-MeWV6rrGiBMjx3auXLF%2Fimage.png?alt=media\&token=3c838719-1113-4e33-9c5a-7e6f1be94eb9)

All going well, you should start seeing data populate the dashboard.

{% hint style="info" %}
Note: you man need to adjust the time range in the top right of the window to a few minutes until enough data has been collected.
{% endhint %}

![Node Exporter Full Dashboard](https://3418161995-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MeGGhDBZUaXZ9G17tnc%2F-MeWTc3Q_o7cIHNKwNxd%2F-MeWVNKlInujvXSWDlJs%2Fimage.png?alt=media\&token=2d1ea9cd-45e3-4d73-b9b1-00dbe705bc6c)

#### Radix Node Dashboard (Optional)

![Betanet Radix Node Dashboard](https://3418161995-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MeGGhDBZUaXZ9G17tnc%2F-MeZtlL8c6bva06I1z8_%2F-MeZz0nvXo-QsRBj9LGg%2Fimage.png?alt=media\&token=2d816a47-a751-4c6a-8fce-e9130098c72d)

The Radix Team have provided their own Grafana dashboard which provides individual node and network wide metrics. Because our Node Server runs in a Docker environment there are additional steps we need to perform so that Prometheus can scrape the data within the Docker container. Please head to the link below to configure this dashboard.

{% content-ref url="/pages/-MeZwI\_93D015-sx2Nkg" %}
[Configure Radix Node Dashboard on Grafana](/install-and-configure-the-radix-validator-software/configure-radix-node-dashboard-on-grafana)
{% endcontent-ref %}

## Alerting

In this section we will be configuring alerts for two popular services

1. Telegram - a free chat app
2. PagerDuty - an enterprise level incident management service (with a free tier)

### Telegram

We will configure Grafana to send alerts to a Telegram chat account

#### Create the Telegram bot

First thing to do is to search for 'Botfather' in Telegram.

![Search for BotFather](https://3418161995-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MeGGhDBZUaXZ9G17tnc%2F-MeWpBfO-agOKnHhv36E%2F-MeZOjSeND54ATNRgIOm%2Fimage.png?alt=media\&token=34dd2ecf-360f-4f8c-b05c-900b811863f7)

Enter `/start` to see the list of available commands.&#x20;

```
/start
```

Enter `/newbot` to request a new bot account

```
/newbot
```

Give your bot a name. It must end with 'bo&#x74;*'. eg.* RadixMonitorin&#x67;*\_*&#x62;ot

![Enter /start then /newbot](https://3418161995-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MeGGhDBZUaXZ9G17tnc%2F-MeWpBfO-agOKnHhv36E%2F-MeZOXEkjSGUqACsRfDg%2Fimage.png?alt=media\&token=f564ab3e-979d-4cb9-a176-6f78a49140ad)

You will receive a response with your API token. Keep this secure and safe.

![Your API token](https://3418161995-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MeGGhDBZUaXZ9G17tnc%2F-MeWpBfO-agOKnHhv36E%2F-MeZQMW6MeLoZeTzBXgS%2Fimage.png?alt=media\&token=fb537fcf-3da1-4975-a98d-8f9b8b47c417)

You will also need your Chat ID. Search for 'Chat ID Echo' in Telegram

![Your Chat ID](https://3418161995-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MeGGhDBZUaXZ9G17tnc%2F-MeWpBfO-agOKnHhv36E%2F-MeZPb5kgjWbC29opins%2Fimage.png?alt=media\&token=d53e67f6-7572-4f82-8ebd-0970635da6be)

Enter `/start` and you will receive your Chat ID in return.

```
/start
```

![Chat ID](https://3418161995-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MeGGhDBZUaXZ9G17tnc%2F-MeWpBfO-agOKnHhv36E%2F-MeZPqdCCyFMERfVi9hd%2Fimage.png?alt=media\&token=613b89c9-f388-4539-bdbf-2b9adb7d4a44)

Now head back to Grafana and on the main Grafana homepage head to the Alerting page, select 'Notification channels' Give your Alert a name eg. Telegram. Change the 'Type' to Telegram and enter in your API token and Chat ID.

![Configure the Telegram alerts](https://3418161995-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MeGGhDBZUaXZ9G17tnc%2F-MeWpBfO-agOKnHhv36E%2F-MeZWYoF5yzdJxkdG1hs%2Fimage.png?alt=media\&token=1f4eb33d-2b44-4c04-bf01-a9d5ec7e879f)

Click Save and Test. All going well, you should receive an alert.

![](https://3418161995-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MeGGhDBZUaXZ9G17tnc%2F-MeWpBfO-agOKnHhv36E%2F-MeZWmHiOFmXZoRuEEP5%2Fimage.png?alt=media\&token=595f80ab-0383-4b6d-a171-acf6ec7b41eb)

#### Additional Bot Security

Ok, we have set our bot, now we will add some security options. By default, our bot can be added to different groups, i.e. anyone can add it to a group which we don't really want.

In order to disable this configuration, we open the chat with the BotFather in Telegram and enter:

```
/setjoingroups
```

Select the name of your bot or the bot that you want to change this feature. After selecting it, this message appears:

```
Enable - bot can be added to groups.
Disable - block group invitations, the bot can't be added to groups.
Current status is: ENABLED
```

As you can see the current status of this feature is ENABLED, we will choose the option Disable. The following message is displayed:

```
Success! The new status is: DISABLED.
```

![](https://3418161995-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MeGGhDBZUaXZ9G17tnc%2F-MeWpBfO-agOKnHhv36E%2F-MeZNzm7HISt2EwfFmwc%2Fimage.png?alt=media\&token=dd4ca720-3c22-4289-86c6-619eb2812bf7)

### Create Grafana Alerts

By default the Grafana alerting functionality is not compatible with query variables. Both the Grafana Node Exporter and Radix Node Dashboard use variables so we will need to create copies of any panel we would like to receive alerts from and edit the query, replacing any variables with their actual values. In the example below we will use the Validator 'UP/DOWN' status panel from the Radix Node Dashboard and create a copy of this panel. We need to configure the panel with the following settings

![](https://3418161995-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MeGGhDBZUaXZ9G17tnc%2F-MfcbCh3MHUPUG2HbmbK%2F-MfccQaQB47M7H-ERyH4%2Fimage.png?alt=media\&token=ab183330-48a4-4f7d-ab29-4f8f733c3b88)

The important bit is to edit the metrics field to specify your node IP

![](https://3418161995-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MeGGhDBZUaXZ9G17tnc%2F-MfcbCh3MHUPUG2HbmbK%2F-Mfcclk14ylHvyIRY7By%2Fimage.png?alt=media\&token=2db9798a-3671-4b4a-9844-b5c740b51922)

If we replace any `instance` variables with `<NODE-SERVER-IP>:443` the Alert tab will appear and we can configure our alert.

![](https://3418161995-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MeGGhDBZUaXZ9G17tnc%2F-MeWpBfO-agOKnHhv36E%2F-MeZaJyHbNkX0Obju5oq%2Fimage.png?alt=media\&token=0768c7e5-0422-40db-a4e0-be6e7cc783f5)

Click the 'Alert' tab and&#x20;

* Give the rule a name
* Set the conditions. For this rule we can use WHEN sum() is below 1
* Configure the error handling as desired
* Send the notifications to Telegram and/or PagerDuty
* Include a message
* Save the Dashboard

![](https://3418161995-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MeGGhDBZUaXZ9G17tnc%2F-MeWpBfO-agOKnHhv36E%2F-MeZcdA3k6Z394H4udRg%2Fimage.png?alt=media\&token=31067e83-594d-482c-a722-51ef894579ea)

The first 'No Data' alert will be received almost instantaneously.&#x20;

![No Data alert](https://3418161995-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MeGGhDBZUaXZ9G17tnc%2F-Meo1WfzqNWm9KyI1q9x%2F-Meuuojvm5epXmaQ6rmz%2Fimage.png?alt=media\&token=5b607bb1-c634-4baf-99c6-39b4826d504a)

The 'Down' alert will take approx 10 mins to trigger due to Grafana going through two steps.

1. An Amber stage which will start 5 mins after the server goes offline, this will last for 5 mins.
2. Therefore 10 mins after the server goes offline a Red alert will trigger and send the notification.&#x20;

If we bring the server down and wait for 10 mins we should see the following alert in Telegram. Bringing the server back up again will send the second alert to notify the server is now 'OK'

![](https://3418161995-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MeGGhDBZUaXZ9G17tnc%2F-MgyUA2VqYmELZpgsJzx%2F-MgzaXG8cqchaEqvjVR6%2Fimage.png?alt=media\&token=d477e500-1142-4467-a884-fdde0616dd53)

Following the same procedure similar alerts can be created for CPU, RAM, Disk, etc set to trigger an alert if they fall below or exceed a certain threshold. eg. when CPU or RAM is above 75% utilisation.

![](https://3418161995-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MeGGhDBZUaXZ9G17tnc%2F-MeZtlL8c6bva06I1z8_%2F-Me_-mFIHGudJ0qtOpmP%2Fimage.png?alt=media\&token=2b86eb2f-5549-470c-9bab-6cbb32a68a9f)

### PagerDuty

Pagerduty is a cloud hosted incident management platform that integrates with many existing alerting and incident management services. They offer a free tier with limited functionality which we will use to begin with. The additional benefit with PagerDuty is they offer SMS, email and phone call notifications.

There are two key steps

1. Sign-up for an account
2. Register a new 'Service' and integrate with Grafana (using their Prometheus integration key). Their Prometheus Integration guide is [here](https://www.pagerduty.com/docs/guides/prometheus-integration-guide/). Note: we only need the Integration key so we can copy that to the Grafana alerting section.

Once you have created a PagerDuty free tier account [here](https://www.pagerduty.com/) create a new 'Service'.

![](https://3418161995-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MeGGhDBZUaXZ9G17tnc%2F-MeWpBfO-agOKnHhv36E%2F-MeZT5UtGg07727R9Kdg%2Fimage.png?alt=media\&token=b51f1ce2-4cef-4159-8384-cd574f0d1b43)

Select the default 'Escalation Policy'

![](https://3418161995-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MeGGhDBZUaXZ9G17tnc%2F-MeWpBfO-agOKnHhv36E%2F-MeZTInOQsggLHSGkzJ4%2Fimage.png?alt=media\&token=a24238e5-ead7-4ba3-9241-80ab99f1f33e)

Select the default 'Intelligent' alert grouping

![](https://3418161995-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MeGGhDBZUaXZ9G17tnc%2F-MeWpBfO-agOKnHhv36E%2F-MeZTla6iJiiesDY70Q-%2Fimage.png?alt=media\&token=09785613-1054-4bd4-bcb5-63ea09b0dcf4)

Search for 'Prometheus' at the integrations step.

![Prometheus Integration](https://3418161995-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MeGGhDBZUaXZ9G17tnc%2F-MeWpBfO-agOKnHhv36E%2F-MeZU3ZyIqhJVXeTawP5%2Fimage.png?alt=media\&token=dcf2035b-40e1-466b-b022-f7ddf75bc8b2)

Copy the 'Integration Key' and save it somewhere safe.

![](https://3418161995-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MeGGhDBZUaXZ9G17tnc%2F-MeWpBfO-agOKnHhv36E%2F-MeZUi8X9scj7O7G6lOT%2Fimage.png?alt=media\&token=8878d6e1-d28f-4432-94c1-a961758738d3)

Now click on your Profile Avatar in the top right hand corner and select 'My Profile'. In the 'Contact Information' tab add your mobile number and/or an email address to receive SMS, Phone call and email alerts as desired.

Once complete, head back to Grafana and create another 'Notification Channel' for Pager Duty. Add your Integration Key, then save and test.

![Grafana Pager Duty Alerting](https://3418161995-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MeGGhDBZUaXZ9G17tnc%2F-MeWpBfO-agOKnHhv36E%2F-MeZVcN8ElESrlpIyCOW%2Fimage.png?alt=media\&token=7724303e-ce4c-46bb-a070-949d6bc090e3)

You should receive an alert via your chosen channels (SMS/Phone/Email) and your PagerDuty Incident dashboard should also update.

![PagerDuty Incident Dashboard](https://3418161995-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MeGGhDBZUaXZ9G17tnc%2F-MeWpBfO-agOKnHhv36E%2F-MeZW0IHC_ew9JxhPca3%2Fimage.png?alt=media\&token=25aaafbd-1949-40a3-95de-0ca71689f154)

That's it! You have completed all the steps for adding monitoring and alerting of your Radix validator node

### Additional Sources

{% embed url="<https://blog.timescale.com/blog/grafana-101-getting-started-with-alerting-recap-and-resources/>" %}

{% embed url="<https://blog.smuts.me/grafana-telegram-alerts/>" %}

{% embed url="<https://www.lvlup-stakepool.com/monitoring/alert_telegram.html>" %}


# SSL Certificate with Certbot and Let's Encrypt

## Introduction

For this section we will assign a domain name to our Monitoring server and install a free Let's Encrypt SSL certificate using Certbot

{% hint style="info" %}
**Note:** In a later tutorial we will negate the need for this certificate by sending traffic through Cloudflare which assigns their own SSL certificate.
{% endhint %}

## Pre-requisites

* To assign a Domain name you will need to request a domain name through a Domain registrar.
*

## NGINX

## Certbot

<https://certbot.eff.org/lets-encrypt/ubuntufocal-nginx>


# Install the Radix Node

## Introduction

This section of the guide will walk you through the steps to complete a Docker versioned install of the Radix Node Runner package. Alternatively, Radix also provide the instructions for a Systemd install. The original installation instructions for both can be found here;

{% embed url="<https://docs.radixdlt.com/main/index.html>" %}

## Pre-requisites

Port 30000/tcp opened for gossip

```
sudo ufw allow 30000/tcp
```

Port 443/tcp opened for NGINX

```
sudo ufw allow 443/tcp
```

{% hint style="warning" %}
The CLI has been developed with Ubuntu 20.04 and is not compatible with Windows
{% endhint %}

## Install the CLI and then install the Docker Node version using the CLI

Radix have developed a CLI tool that facilitates an easier method for installing the package and for later querying the node. We will cover the querying elements in a later section.

Make sure all your system packages are up-to-date.

```
sudo apt update -y
```

ensure **wget** is installed

```
sudo apt install wget
```

### Download the CLI package

Go to the URL <https://github.com/radixdlt/node-runner/releases> and look for the entry with the **Latest release** marker and copy the link to the latest release.

```
wget -O radixnode https://github.com/radixdlt/node-runner/releases/download/1.0.4/radixnode-ubuntu-20.04
```

&#x20;Set the permissions on the script to executable.

```
sudo chmod +x radixnode
```

&#x20;move it to the `/usr/local/bin/` directory

```
sudo mv radixnode /usr/local/bin
```

### Install Docker&#x20;

```
radixnode docker configure
```

log out of your ssh session and back in again

```
Exit ssh login and relogin back for user addition to group "docker" to take effect
```

### Configure NGINX

```
radixnode auth set-admin-password --setupmode DOCKER
```

create and enter a strong NGINX `admin` password

set the NGINX password environment variable

```
echo 'export NGINX_ADMIN_PASSWORD="nginx-password"' >> ~/.bashrc
```

add it to your session

```
source ~/.bashrc
```

do the same for a NGINX `superadmin` password

```
radixnode auth set-superadmin-password --setupmode DOCKER
```

set the NGINX superadmin environment variable

```
echo 'export NGINX_SUPERADMIN_PASSWORD="nginx-password"' >> ~/.bashrc
```

add it to your session

```
source ~/.bashrc
```

### Install and start the Radix Docker node

If you are building a validator node the enter a switch for `-n fullnode`**.** Alternatively you would enter `-n archivenode` for a non-validating archive node.

```
radixnode docker setup  \
     -n fullnode \
     -t radix://tn1qgtjz4432e7ghdfsdc6ssu4v63rwxk2g84e4yedadeasfsfaw3995l9gl6l@52.215.119.169
```

the second `-t` switch is for the seed node that you use to connect to the network. You would normally choose a node closest to your hosting region.

| <p></p><p>Asia Pacific (south-east)</p>                                                 |
| --------------------------------------------------------------------------------------- |
| <radix://tn1qdg87mk4gr8dlmfv82y9cqleqkcjm0fxlxe8vhcxvskd3k90jcmgk4ewsmw@13.210.91.116>  |
| <radix://tn1qfzy56n7wsspqjn8kal8n4ls9rs6lth3x6fzqhhdgqa7emwqz9w8j0xp4e3@3.106.19.89>    |
| <radix://tn1qt86e6vjvlcvmhgcgeu4szkmd70kg54g0j2fef9ttpt738xp0jjlv693c92@54.206.193.154> |

| Asia Pacific (south)                                                                   |
| -------------------------------------------------------------------------------------- |
| <radix://tn1qvgnymu94dw5gfug2e678x9h0puupk8vkm0z0083744jscukqvgsxjpgrnr@3.108.246.15>  |
| <radix://tn1qdhf7aeat739n77e227l3vgvv56kntp20ennaj38t5jc8988r5s2zxzxp77@3.108.209.244> |
| <radix://tn1qwrerrj64fc66v8ztcgd2fr8szx73k7ermkg5hyy8q7qelnu48svw44m362@3.108.183.207> |

| <p></p><p>EU (west)</p>                                                                 |
| --------------------------------------------------------------------------------------- |
| <radix://tn1qt9kqzzqyj27zv4n67f2jrzgd24hsxfwe8d4kw9j4msze7rpdg3guvk07jy@54.76.86.46>    |
| <radix://tn1qw3dyujr8ss29648wfqzenwx7el08yj6mpru6z62hxzlh9n3sgwwqdkl295@52.16.242.94>   |
| <radix://tn1qgtjz4432e7ghdfsdc6ssu4v63rwxk2g84e4yedadeasfsfaw3995l9gl6l@52.215.119.169> |

| <p></p><p>US (east)</p>                                                                |
| -------------------------------------------------------------------------------------- |
| <radix://tn1qvluh8d3e6uxnm2k0h6zfnng0r7hkgcd8ppjn8slzyhs7vqlavw0va9de6q@3.222.172.90>  |
| <radix://tn1q0g9zpvv2ggw99fz86q52csdjk2t0ynqz0r0e23xhdszx3r7sv9jyx84lc4@54.210.93.129> |
| <radix://tn1qt2rma6397uytnusr0ct777tp3gnzkh65apnmhz9zqjqvmwjxwvhv8rmdd3@34.195.139.75> |

It will now go through a list of questions to configure your node

![](https://3418161995-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MeGGhDBZUaXZ9G17tnc%2F-Mez6jR9t2tJRn9Z8zz_%2F-MezNSpXm5wT7KWMUh3A%2Fimage.png?alt=media\&token=d0465d40-5047-4e1d-a8ef-7e266e37c042)

enter **y** at the prompt

```
Going to setup node type fullnode for version 1.0-beta.37 from location https://github.com/radixdlt/radixdlt/releases/download/1.0-beta.37/radix-fullnode-compose.yml.

Do you want to continue [Y/n]?:
```

as this is our first install enter **n** for the node keystore file.

```
Do you have keystore file named 'node-keystore.ks' already from previous node Y/n?:n
```

Enter a strong password for your keystore file

```
Enter the password of the new file 'node-keystore.ks':
```

Choose **Y** to configure the data directory.

```
Do you want to configure data directory for the ledger [Y/n]?:Y
```

Enter the absolute path to where you want to store the Radix DB

```
Enter the absolute path to data DB folder: /home/radixdlt/RADIXDB
```

enter **s** for stokenet or **m** for mainnet

```
Enter the network you want to connect [S]Stokenet or [M]Mainnet or network_id:s
```

the install is now complete. Your docker yml file will be displayed where you can check the docker settings. Enter **y** to start your node

![](https://3418161995-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MeGGhDBZUaXZ9G17tnc%2F-MeymY0PG4AidrvUA_KB%2F-MeypIJ___-PPtkPq-qp%2Fimage.png?alt=media\&token=e9c09d29-8bbb-43fc-b8e2-73c01143e343)

Your node is node installed. Before we move to the next section we'll confirm some basic checks

```
radixnode api account get-info
```

should return the following Json with your own **tdx**... node wallet address

```
{"result": {"address": "tdx1qsprd6tt0dxvw9qks2mh743tr68s4ft9p6qv2zkfc622uymczragyfclqqu7e", "balance": {"stakes": [], "tokens": []}}, "id": 1, "jsonrpc": "2.0"}
```

check the software version is correct

```
radixnode api version

{"version": "1.0-beta.37-release~1.0-beta.37-SNAPSHOT"}
```

check the node's health

```
radixnode api health

{"status": "UP"}
```

other reponses may include

* BOOTING - node is booting and not ready to accept requests
* SYNCING - node is catching up the network
* UP - node is in sync with consensus
* STALLED - node is out of sync and not trying to sync with network, but network is still available.
* OUT\_OF\_SYNC - node is out of sync and does not get updates from network (for example, connection to network is lost).

the node is now ready to be registered on the network.

### Optimise Node

You need to install ansible first

```
sudo apt install ansible
```

### Original Radix Docs

Docker Install using CLI <https://docs.radixdlt.com/main/node/cli-install-node-docker.html>

SystemD Install using CLI <https://docs.radixdlt.com/main/node/cli-install-node-systemd.html>


# Register as a Validator

## Introduction

The following section will take you through the steps to register your node as a Validator node on the network

## Pre-requisites

* 30 XRD in your desktop wallet to be able to send to the node's wallet for registration
* radixnode cli is installed

## Registration

Begin by sending some XRD to your node's wallet address so that there is enough to cover the registration fees. 5-10 XRD is plenty.

Check your node's wallet address

```
radixnode api account get-info

{"result": {"address": "tdx1qsprd6tt0dxvw9qks2mh743tr68s4ft9p6qv2zkfc622uymczragyfclqqu7e",
```

in this instance the wallet address begins tdx...&#x20;

```
tdx1qsprd6tt0dxvw9qks2mh743tr68s4ft9p6qv2zkfc622uymczragyfclqqu7e
```

head to your desktop or Ledger wallet and send the XRD to that address, then check it's been received by entering

```
radixnode api account get-info
```

```
Confirm the 'amount' paramater has a value

{"result": {"address": "tdx1qsprd6tt0dxvw9qks2mh743tr68s4ft9p6qv2zkfc622uymczragyfclqqu7e", "balance": {"stakes": [], "tokens": [{"amount": "10000000000000000000", "rri": "01"}]}}, "id": 1, "jsonrpc": "2.0"}
```

enter the following command to register as a validator

```
radixnode api account update-validator-config
```

enter **true** add then add your **name**, **url** and **fee** and whether you'll **accept delegations**. You'll need to specify a wallet address of where emissions should go to. It is recommended that this is an external wallet address such as the upcoming Ledger wallet or Desktop wallet and not your node's wallet address.

![](https://3418161995-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MeGGhDBZUaXZ9G17tnc%2F-Mez6jR9t2tJRn9Z8zz_%2F-Mez_UbEmCG9wqtpSdPM%2Fimage.png?alt=media\&token=a629ee2e-a9b6-4dc2-a227-645feaebdbc2)

check the registration was successful

```
radixnode api validation get-validator-info
```

```
{"result": {"owner": "tdx1qspy3hhj2xzr9vcwfkw45lclj6kdkpu7xpueyl3ym32lcwlcg2lm9hcw3gftc", \ 
"address": "tv1qgmwj6mmfnr3g95zkal4v2c73u922egwsrzs4jwxjjhpx7qsl2pzwa53mte", \
"stakes": [], "allowDelegation": true, \ 
"name": "Avaunt Staking 03", "validatorFee": "2.0", \
"registered": true, "totalStake": "0", "url": "https://avaunt-staking.com"}, "id": 1, "jsonrpc": "2.0"}

```

### How to update the Validator

```
radixnode api account update-validator-config
```

### How to unregister your Validator

```
radixnode api account update-validator-config
```

change the value to **false**

If you have delegates signed on to your validator node, then you must inform them before you unregister the node.

##


# Stop or Start your node

### **How to stop your node**

```
radixnode docker stop -f radix-fullnode-compose.yml
```

### How to start your node

change the value for the `-t` switch to the same value you used when setting up your node

```
radixnode docker start \
        -f radix-fullnode-compose.yml \
        -t radix://tn1qgtjz4432e7ghdfsdc6ssu4v63rwxk2g84e4yedadeasfsfaw3995l9gl6l@52.215.119.169
```


# Update your Validator

## How to Update your node

```
radixnode docker setup  \
     -u \
     -n fullnode \
     -r 1.0-beta.37 \
     -t radix://tn1qgtjz4432e7ghdfsdc6ssu4v63rwxk2g84e4yedadeasfsfaw3995l9gl6l@52.215.119.169
```

The `-u` option specifies that this launch of the node will be an update, causing radixnode to create a backup of the current configuration file and ensure that the node has stopped before applying the changes.&#x20;

The `-n` option sets the node type, which can be either fullnode or archivenode.&#x20;

The `r` option is optional and specifies the release of the node software you wish to install. If not provided it will use the latest release from <https://github.com/radixdlt/radixdlt/releases>.&#x20;

`-t` is the IP address of a radix node that you can use to join the network.&#x20;

Again, select a seed node closest to where your host is located.

the third `-t` switch is for the seed node that you use to connect to the network. You would normally choose a node closest to your hosting region.

| <p></p><p>Asia Pacific (south-east)</p>                                                 |
| --------------------------------------------------------------------------------------- |
| <radix://tn1qdg87mk4gr8dlmfv82y9cqleqkcjm0fxlxe8vhcxvskd3k90jcmgk4ewsmw@13.210.91.116>  |
| <radix://tn1qfzy56n7wsspqjn8kal8n4ls9rs6lth3x6fzqhhdgqa7emwqz9w8j0xp4e3@3.106.19.89>    |
| <radix://tn1qt86e6vjvlcvmhgcgeu4szkmd70kg54g0j2fef9ttpt738xp0jjlv693c92@54.206.193.154> |

| Asia Pacific (south)                                                                   |
| -------------------------------------------------------------------------------------- |
| <radix://tn1qvgnymu94dw5gfug2e678x9h0puupk8vkm0z0083744jscukqvgsxjpgrnr@3.108.246.15>  |
| <radix://tn1qdhf7aeat739n77e227l3vgvv56kntp20ennaj38t5jc8988r5s2zxzxp77@3.108.209.244> |
| <radix://tn1qwrerrj64fc66v8ztcgd2fr8szx73k7ermkg5hyy8q7qelnu48svw44m362@3.108.183.207> |

| <p></p><p>EU (west)</p>                                                                 |
| --------------------------------------------------------------------------------------- |
| <radix://tn1qt9kqzzqyj27zv4n67f2jrzgd24hsxfwe8d4kw9j4msze7rpdg3guvk07jy@54.76.86.46>    |
| <radix://tn1qw3dyujr8ss29648wfqzenwx7el08yj6mpru6z62hxzlh9n3sgwwqdkl295@52.16.242.94>   |
| <radix://tn1qgtjz4432e7ghdfsdc6ssu4v63rwxk2g84e4yedadeasfsfaw3995l9gl6l@52.215.119.169> |

| <p></p><p>US (east)</p>                                                                |
| -------------------------------------------------------------------------------------- |
| <radix://tn1qvluh8d3e6uxnm2k0h6zfnng0r7hkgcd8ppjn8slzyhs7vqlavw0va9de6q@3.222.172.90>  |
| <radix://tn1q0g9zpvv2ggw99fz86q52csdjk2t0ynqz0r0e23xhdszx3r7sv9jyx84lc4@54.210.93.129> |
| <radix://tn1qt2rma6397uytnusr0ct777tp3gnzkh65apnmhz9zqjqvmwjxwvhv8rmdd3@34.195.139.75> |


# Configure Radix Node Dashboard on Grafana

## Introduction

By default the Radix team have provided their own Prometheus server and Grafana Dashboard within the node build. However you may want to consolidate dashboards with other systems or services so the steps below provide a guide on how to import the Radix node metrics into your own monitoring server.

### Pre-Reqs

{% content-ref url="/pages/-MeGpfI1kydtRDQUi2M4" %}
[Build and Configure the Monitoring Webserver](/build-the-webserver)
{% endcontent-ref %}

### Pre-Configure the Node Server

To export the metrics you will need to install Prometheus on the Node server. The easist way to do this is to install it via the Radixnode CLI. You can follow the instructions [here](https://docs.radixdlt.com/main/node/install-grafana-dashboard.html).<br>

Then open the firewall on the Node Server if you haven't already

```bash
sudo ufw allow proto tcp from <WebServer IP> to any port 8099
```

### Configure your Monitoring server

On your Monitoring WebServer copy and paste the following into the existing yml file. From the Monitoring Build guide this was created at `/etc/prometheus/prometheus.yml`

{% code title="/etc/prometheus/prometheus.yml" %}

```bash
  - job_name: 'mynode'
    scheme: https
    basic_auth:
      username: "metrics"
      password: "<YOUR METRICS PASSWORD>"
    tls_config:
      insecure_skip_verify: true
    static_configs:
      - targets:
          - <NODE IP 1>
          - <NODE IP 2>
```

{% endcode %}

Save and close the file then restart the service

```bash
sudo systemctl daemon-reload
sudo systemctl restart prometheus
```

### Import the Radix Node Runner Dashboard JSON&#x20;

This can either be exported by running the `radixnode monitoring setup` CLI command and accessing the dashboard from `http://<YOUR NODE IP>:3000/d/radix_node_dashboard/radix-node-dashboard?orgId=1&refresh=5s`

then export the Dashboard JSON from there or ask one of the other node runners to export it for you.

### Configure the Dashboard

Depending on what other infrastructure you are monitoring with Grafana you may need to specify the particular node IP for some of the panels.

Eg. you may need to edit the Status panel to focus on the node IP.

![](https://3418161995-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MeGGhDBZUaXZ9G17tnc%2F-MfcIoeDQcEwcjuhB1EK%2F-MfcLum7Aes704mQVEoI%2Fimage.png?alt=media\&token=bdd52ed2-66c9-4fa0-b914-abd05f259dc6)

and that should be it.

You can now go ahead and add alerting for any of the panels eg. create a Telegram alert when the Node status is 'Down'.

{% content-ref url="/pages/-MeJpaK7z\_CvKVo4EQ7n" %}
[Monitoring and Alerting](/monitoring-and-alerting)
{% endcontent-ref %}