Skip to content

Incident Summary: GCP Patch Job Failure on TMS production server

Summary of the issue

  • Symptoms: GCP OS Config patch failed with exit status 100 during the APPLY_PATCHES task execution.

  • Root Cause: The repository metadata for google-cloud-ops-agent-jammy-2 changed its upstream Origin and Label properties from google-cloud-ops-agent-jammy to google-cloud-ops-agent-jammy-2. APT's security controls (apt-secure) blocked non-interactive package operations until this change was explicitly accepted by an administrator.

Commands executed

  • System package diagnostic and repair
# Clear any interrupted package configurations or locked installations
sudo dpkg --configure -a
sudo apt-get install -f

# Clear local package index caches
sudo apt-get clean

# Explicitly accept repository metadata changes and refresh package lists
sudo apt-get update --allow-releaseinfo-change

# Restart the OS Config agent daemon to apply changes
sudo systemctl restart google-osconfig-agent

# Verify agent status (Verify: active (running))
sudo systemctl status google-osconfig-agent

Retrigering the patch

gcloud compute os-config patch-jobs execute \
    --project="attracs-services-195009" \
    --instance-filter-names="zones/<VM_ZONE>/instances/tms-confidential-instance-stable-2"

Verification: Monitored execution until state changed to SUCCEEDED.