Skip to content

Exporting Metrics (Observability configuration)

This documentation outlines the custom metric pipeline implemented to track pending OS updates, security patches, reboot requirements, and system uptime across our virtual machines, allowing the data to be visualized seamlessly in Grafana.

  1. VM Reporting Script Setup

  2. A custom Bash script is deployed to each VM to gather local package manager statistics and output them in standard Prometheus OpenMetrics format.

  3. Location: The script is stored on the virtual machines at the absolute path /root/scripts/report-apt.sh.

  4. Mechanism: It utilizes the native /usr/lib/update-notifier/apt-check tool to parse pending updates and checks the presence of /var/run/reboot-required.

  5. Textfile Output: The script formats the parsed data into Prometheus gauge metrics with an explicitly declared host="$HOSTNAME" label and writes it directly to the Node Exporter textfile collector directory.

  6. Atomic Writing: Data is written to a temporary .tmp file first and then moved to prevent Node Exporter from reading partially written files. 

#!/bin/bash
TEXTFILE_DIR="/var/lib/node_exporter/textfile_collector"
TEMP_FILE="${TEXTFILE_DIR}/apt_check.prom.tmp"
PROM_FILE="${TEXTFILE_DIR}/apt_check.prom"

mkdir -p "$TEXTFILE_DIR"

APT_CHECK=$(/usr/lib/update-notifier/apt-check 2>&1)
UPDATES=$(echo "$APT_CHECK" | cut -d ';' -f 1)
SECURITY=$(echo "$APT_CHECK" | cut -d ';' -f 2)
REBOOT=$([ -f /var/run/reboot-required ] && echo 1 || echo 0)

cat <<EOF > "$TEMP_FILE"
# HELP host_apt_pending_updates Total number of pending package updates
# TYPE host_apt_pending_updates gauge
host_apt_pending_updates{host="$HOSTNAME",env="stable"} $UPDATES

# HELP host_apt_security_pending_updates Total number of pending security updates
# TYPE host_apt_security_pending_updates gauge
host_apt_security_pending_updates{host="$HOSTNAME",env="stable"} $SECURITY

# HELP host_apt_reboot_required Indicates whether a system reboot is required (1 or 0)
# TYPE host_apt_reboot_required gauge
host_apt_reboot_required{host="$HOSTNAME",env="stable"} $REBOOT
EOF

mv "$TEMP_FILE" "$PROM_FILE"
** Note: in the previous example, the script is setup to be run in a stable machine. If you were to run this in nightly make sure you change the name of the environment to nightly before running it.


Node Exporter Docker Container Setup To expose the script's metrics alongside standard host hardware metrics, Node Exporter is run via Docker with textfile collection enabled.

  • Directory Creation: The host textfile directory is created at /var/lib/node_exporter/textfile_collector.

  • Host Networking: The container uses --net="host" to bind directly to the host's port 9100.

  • Volume Mount: The textfile directory is mounted into the container at /etc/node_exporter alongside the root filesystem mount.

  • Remove the old node exporter first.

sudo docker stop node-exporter && sudo docker rm node-exporter


mkdir -p /var/lib/node_exporter/textfile_collector

docker run -d \
  --name node-exporter \
  --net="host" \
  --pid="host" \
  -v "/:/host:ro,rslave" \
  -v "/var/lib/node_exporter/textfile_collector:/etc/node_exporter" \
  quay.io/prometheus/node-exporter:v0.18.1 \
  --path.rootfs=/host \
  --collector.textfile.directory=/etc/node_exporter

Automation (Cron Schedule)

To ensure Grafana dashboards always reflect current data, the reporting script is configured to run automatically without manual intervention.

  • Permissions: The file is granted executable permissions using the sudo chmod +x /root/scripts/report-apt.sh command.

  • Crontab Integration: The automation is handled entirely by the root user's task scheduler, accessed via sudo crontab -e.

  • Frequency: The execution schedule is set to 0 0 * * * /root/scripts/report-apt.sh, guaranteeing the metrics update nightly at exactly midnight.

Prometheus & Grafana Table Configuration

Grafana queries Prometheus directly, combining custom APT metrics with standard Node Exporter system metrics into a multi-column table.

  • Metric Selection: Query A targets host_apt_reboot_required and Query B targets max by (instance) (time() - node_boot_time_seconds) for VM uptime.

  • Query Format: Both queries are formatted as Table with type set to Instant to eliminate repeating time-series rows.

  • Data Merging: An Inner join transformation on the field instance is applied to merge reboot status and uptime into single host rows while filtering out unmonitored metrics.

  • Field Organization: An Organize fields transformation hides metadata columns (Time, job, name, instance) and renames headers to Host, Reboot Status, and Uptime.

  • Specific Column Overrides: A field override on Reboot Status maps values (0 -> OK in green, 1 -> REBOOT REQUIRED in red), while a separate override on Uptime sets the unit to Time -> Duration (s) to render human-readable uptime without affecting non-target columns.