Skip to content

Incident Summary: GCP Patch Job Failure on Production Server

Affected Instance: road-confidential-instance-stable-1

  • Initial Symptom: Google Cloud OS Patch Management jobs were failing with "COMPLETED_WITH_ERRORS" and "exit status 100", resulting in empty or generic error logs in the Google Cloud Console.

Root Cause Analysis

The patching failures were caused by two sequential issues at the OS package manager (apt/dpkg) level, which prevented the background OS Config Agent from executing automated updates:

  • Corrupted & Unverified Package Repositories: The agent could not refresh the list of available updates (apt-get update) due to a 400 Bad Request from a corrupted Google Cloud SDK local cache, and a NO_PUBKEY error from an outdated HashiCorp repository configuration.

  • Interactive Configuration Prompts: Once the repository lists were repaired, the subsequent automated installation attempt hung. The base-files package update detected custom modifications to your company's login banners (/etc/issue and /etc/issue.net). Because automated patch agents run without human interaction, the agent could not answer the prompt asking whether to overwrite these files or keep the current versions, causing the installation to abort.

Actions Taken & Resolution

The following steps were executed safely on the live production server without causing downtime to running applications:

  • Repository Cache Cleared: Executed apt-get clean and cleared the /var/lib/apt/lists/* directory to resolve the 400 Bad Request errors from the Google Cloud CDN.

  • GPG Keys Updated: Downloaded the correct HashiCorp GPG key to the modern /usr/share/keyrings/ directory and updated the sources.list.d configuration, resolving the signature verification errors.

  • Interactive Prompts Resolved: Executed dpkg --configure -a to manually resume the stalled installation. Selected N for both /etc/issue and /etc/issue.net to preserve your custom company banners.

  • Manual Patching Completed: Ran apt-get upgrade to successfully install all 101 pending package updates, including core system utilities, Docker components, and the Linux kernel.

Current Status & Recommendations

The server road-confidential-instance-stable-1 is now fully patched, and the local package manager is completely healthy. The Google Cloud OS Config Agent will automatically detect this healthy state and report back to the GCP Console.

Pending Action (Reboot):

Because the Linux kernel was upgraded during the final manual step (linux-image-6.8.0-1070-gcp), the server requires a standard reboot for the new kernel changes to take effect. The reboot can be performed during the next scheduled maintenance window.

Preventative Measure:

To ensure future Google Cloud patch jobs do not fail when encountering similar customized configuration files, it is highly recommended to run the following command. This forces the package manager to safely keep your modified files by default during automated runs:

echo 'Dpkg::Options {
  "--force-confdef";
  "--force-confold";
}' | sudo tee /etc/apt/apt.conf.d/99force-conf > /dev/null