Post update checks

Post update script checks

The following script evaluates the status of the daemons, process managers, disk space, and boot logs, outputting a color-coded PASS, FAIL, or WARN for each category.

#!/bin/bash

# Ensure script is run as root
if [ "$EUID" -ne 0 ]; then
  echo "Please run as root (e.g., sudo ./post_update_check.sh)"
  exit 1
fi

echo "========================================"
echo " Post-Reboot Health Check Report"
echo "========================================"
echo ""

# Color codes
PASS="[\033[32mPASS\033[0m]"
FAIL="[\033[31mFAIL\033[0m]"
WARN="[\033[33mWARN\033[0m]"

# 1. Check Daemons
echo -n "Checking Supervisord daemon... "
if systemctl is-active --quiet supervisord; then
    echo -e "$PASS"
else
    echo -e "$FAIL"
fi

echo -n "Checking Docker daemon... "
if systemctl is-active --quiet docker; then
    echo -e "$PASS"
else
    echo -e "$FAIL"
fi

# 2. Check Supervisor Apps
echo -n "Checking Supervisor apps state... "
# Extract any app that is NOT "RUNNING"
FAILED_APPS=$(supervisorctl status 2>/dev/null | awk '{if ($2 != "RUNNING") print $1 " (" $2 ")"}')
if [ -z "$FAILED_APPS" ]; then
    echo -e "$PASS"
else
    echo -e "$FAIL"
    echo "  -> Issues detected with: $FAILED_APPS"
fi

# 3. Check Docker Containers
echo -n "Checking Docker containers state... "
# Find containers that are not explicitly "Up"
FAILED_CONTAINERS=$(docker ps -a --format '{{.Names}} ({{.Status}})' | grep -v 'Up ')
if [ -z "$FAILED_CONTAINERS" ]; then
    echo -e "$PASS"
else
    echo -e "$WARN (Some containers are down)"
    echo "  -> $FAILED_CONTAINERS"
fi

# 4. Check Disk Space on Root (/)
echo -n "Checking root disk space (< 90%)... "
DISK_USAGE=$(df / | awk 'NR==2 {print $5}' | sed 's/%//')
if [ "$DISK_USAGE" -lt 90 ]; then
    echo -e "$PASS (${DISK_USAGE}%)"
else
    echo -e "$FAIL (${DISK_USAGE}%)"
fi

# 5. Check System Error Logs since boot
echo -n "Checking critical boot errors... "
# Count priority 3 (err) messages since current boot (-b)
ERROR_COUNT=$(journalctl -p 3 -xb --quiet 2>/dev/null | wc -l)
if [ "$ERROR_COUNT" -eq 0 ]; then
    echo -e "$PASS (0 errors)"
else
    echo -e "$WARN ($ERROR_COUNT error log entries found)"
    echo "  -> Run 'sudo journalctl -p 3 -xb' to review."
fi

echo ""
echo "========================================"
echo " System Uptime: $(uptime -p)"
echo " Kernel Version: $(uname -r)"
echo "========================================"

Script is saved to /root/scripts/post_update_check.sh and permissions of the file updated to chmod +x /root/scripts/post_update_check.sh.

Example output:

./post_update_check.sh
========================================
 Post-Reboot Health Check Report
========================================

Checking Supervisord daemon... [PASS]
Checking Docker daemon... [PASS]
Checking Supervisor apps state... [FAIL]
  -> Issues detected with: road-feedbacks.attracs.com (FATAL)
Checking Docker containers state... [PASS]
Checking root disk space (< 90%)... [PASS] (39%)
Checking critical boot errors... [WARN] (73 error log entries found)
  -> Run 'sudo journalctl -p 3 -xb' to review.

========================================
 System Uptime: up 7 hours, 41 minutes
 Kernel Version: 6.8.0-1069-gcp
========================================