Prometheus and Grafana:
Full Server Monitoring in Under an Hour

Architecture Overview

The Prometheus monitoring stack has three main components:

• Node Exporter - runs on each server you want to monitor, exposes hardware and OS metrics (CPU, memory, disk, network) as an HTTP endpoint on port 9100.
• Prometheus - a time-series database that scrapes (pulls) metrics from your exporters at a configurable interval and stores them. Also handles alerting rules.
• Grafana - a visualisation platform that queries Prometheus and renders dashboards. Can send alerts to Slack, PagerDuty, email, and dozens of other services.

You can run all three on a single monitoring server, or run Node Exporter on each target server and Prometheus/Grafana centrally. For small setups, a single 2 GB RAM server handles hundreds of targets comfortably.

Step 1: Install Node Exporter on Target Servers

Node Exporter needs to run on every machine you want to monitor. The simplest installation uses the official binary:

# Download latest Node Exporter (check releases.prometheus.io for current version)
wget https://github.com/prometheus/node_exporter/releases/download/v1.8.2/node_exporter-1.8.2.linux-amd64.tar.gz
tar xzf node_exporter-1.8.2.linux-amd64.tar.gz
cp node_exporter-1.8.2.linux-amd64/node_exporter /usr/local/bin/

# Create a dedicated system user
useradd --no-create-home --shell /bin/false node_exporter

# Create a systemd service
cat > /etc/systemd/system/node_exporter.service << 'EOF'
[Unit]
Description=Prometheus Node Exporter
After=network.target

[Service]
User=node_exporter
Group=node_exporter
ExecStart=/usr/local/bin/node_exporter
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable --now node_exporter

# Verify it's working
curl -s http://localhost:9100/metrics | head -20

Step 2: Install Prometheus

Prometheus is also distributed as a static binary. Install it on your central monitoring server:

wget https://github.com/prometheus/prometheus/releases/download/v2.53.0/prometheus-2.53.0.linux-amd64.tar.gz
tar xzf prometheus-2.53.0.linux-amd64.tar.gz
cp prometheus-2.53.0.linux-amd64/prometheus /usr/local/bin/
cp prometheus-2.53.0.linux-amd64/promtool /usr/local/bin/

# Create directories
mkdir -p /etc/prometheus /var/lib/prometheus
useradd --no-create-home --shell /bin/false prometheus
chown -R prometheus:prometheus /etc/prometheus /var/lib/prometheus

Create the main configuration at /etc/prometheus/prometheus.yml:

global:
  scrape_interval:     15s
  evaluation_interval: 15s

alerting:
  alertmanagers:
    - static_configs:
        - targets: ['localhost:9093']

rule_files:
  - "alerts.yml"

scrape_configs:
  # Prometheus monitors itself
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  # Your servers - add one entry per host
  - job_name: 'node'
    static_configs:
      - targets:
          - 'server1.example.com:9100'
          - 'server2.example.com:9100'
          - '10.0.0.2:9100'   # via WireGuard VPN

Create the systemd service and start Prometheus:

cat > /etc/systemd/system/prometheus.service << 'EOF'
[Unit]
Description=Prometheus
After=network.target

[Service]
User=prometheus
Group=prometheus
ExecStart=/usr/local/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus \
  --storage.tsdb.retention.time=90d \
  --web.listen-address=127.0.0.1:9090
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable --now prometheus

# Verify scrape targets are healthy
curl -s http://localhost:9090/api/v1/targets | python3 -m json.tool | grep health

Step 3: Install Grafana

# Add Grafana repository
apt install -y apt-transport-https software-properties-common
wget -q -O - https://apt.grafana.com/gpg.key | gpg --dearmor > /etc/apt/trusted.gpg.d/grafana.gpg
echo "deb https://apt.grafana.com stable main" > /etc/apt/sources.list.d/grafana.list

apt update && apt install grafana

systemctl enable --now grafana-server

Grafana starts on port 3000. Log in at http://YOUR_IP:3000 with admin / admin and change the password. Expose it publicly through an Nginx reverse proxy with SSL (see our Nginx guide).

Step 4: Connect Grafana to Prometheus

1. In Grafana, go to Connections → Data Sources → Add data source
2. Select Prometheus
3. Set URL to http://localhost:9090
4. Click Save & Test - it should show "Successfully queried the Prometheus API"

Step 5: Import the Node Exporter Dashboard

The community maintains an excellent pre-built dashboard for Node Exporter metrics. Import it in under a minute:

1. Go to Dashboards → Import
2. Enter dashboard ID 1860 (Node Exporter Full)
3. Select your Prometheus data source
4. Click Import

You now have a complete dashboard showing CPU usage, memory utilisation, disk I/O, network throughput, filesystem usage, and system load - all with historical data and configurable time ranges.

Step 6: Alerting Rules

Create /etc/prometheus/alerts.yml with practical alerting rules:

groups:
  - name: node_alerts
    rules:
      - alert: HighCPUUsage
        expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 85
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "High CPU usage on {{ $labels.instance }}"
          description: "CPU usage is {{ printf \"%.0f\" $value }}% for more than 10 minutes."

      - alert: LowDiskSpace
        expr: (node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}) * 100 < 15
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Low disk space on {{ $labels.instance }}"
          description: "Root filesystem has less than 15% free space."

      - alert: HighMemoryUsage
        expr: (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100 > 90
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "High memory usage on {{ $labels.instance }}"

      - alert: InstanceDown
        expr: up == 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Instance {{ $labels.instance }} is down"

Reload Prometheus: systemctl reload prometheus. For alert delivery (email, Slack, PagerDuty), install and configure Alertmanager - the separate component that routes Prometheus alerts to notification channels.