VM Manager Patching Guide — Taxi Backend (Nightly)

Zero-downtime OS patching strategy for the att-nightly-tms-backend-* servers using Google Cloud VM Manager.

Scope: project nightly environment, zone europe-north1-a
Verified on: 2026-07-31
Dashboard: here

1. Enabling the OS Config API

To use VM Manager for patching, the OS Config API (VM Manager API) must be enabled on the project.

Prerequisites

Requirement	Value
IAM role	`roles/serviceusage.serviceUsageAdmin` (Service Usage Admin) on the project

Option 1 — Google Cloud Console

Open the Google Cloud console and select your project.
Go to the OS Config API page.
Click Enable.

Option 2 — gcloud CLI

gcloud services enable osconfig.googleapis.com --project=PROJECT_ID

Replace PROJECT_ID with your actual Google Cloud project ID.

2. Checking and Installing the OS Config Agent

VM Manager relies on an agent running on each VM to execute patching tasks.

The OS Config agent ships pre-installed on most Google-provided images (CentOS, Debian, Ubuntu, etc.) with a build date of v20200114 or later. Even so, verify it is present and running on each backend server.

Check agent status

gcloud compute ssh INSTANCE_NAME \
  --zone=ZONE \
  --command="sudo systemctl status google-osconfig-agent"

Replace INSTANCE_NAME and ZONE with the details of your backend servers.

A healthy agent reports Active: active (running).

Verification results

`att-nightly-tms-backend-1b-nightly`

gcloud compute ssh att-nightly-tms-backend-1b-nightly \
  --zone=europe-north1-a \
  --tunnel-through-iap \
  --command="sudo systemctl status google-osconfig-agent"

● google-osconfig-agent.service - Google OSConfig Agent
     Loaded: loaded (/lib/systemd/system/google-osconfig-agent.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2026-05-04 01:12:26 UTC; 2 months 27 days ago
   Main PID: 1361210 (google_osconfig)
      Tasks: 15 (limit: 9509)
     Memory: 451.3M
     CGroup: /system.slice/google-osconfig-agent.service
             └─1361210 /usr/bin/google_osconfig_agent

OSConfigAgent Info: Validate: resource "install-pkg" validation successful.
OSConfigAgent Info: Check state: resource "install-pkg" state is COMPLIANT.
OSConfigAgent Info: Policy "goog-ops-agent-policy" resource "wait-for-cloud-init" state: COMPLIANT
OSConfigAgent Info: Policy "goog-ops-agent-policy" resource "add-repo" state: COMPLIANT
OSConfigAgent Info: Policy "goog-ops-agent-policy" resource "install-pkg" state: COMPLIANT
OSConfigAgent Info: Successfully completed ApplyConfigTask
OSConfigAgent Info: Writing inventory to guest attributes   # every ~10 min

Status: ✅ active (running), all goog-ops-agent-policy resources COMPLIANT, inventory reporting on a ~10-minute cadence.

`att-nightly-tms-backend-2b-nightly`

gcloud compute ssh att-nightly-tms-backend-2b-nightly \
  --zone=europe-north1-a \
  --tunnel-through-iap \
  --command="sudo systemctl status google-osconfig-agent"

● google-osconfig-agent.service - Google OSConfig Agent
     Loaded: loaded (/lib/systemd/system/google-osconfig-agent.service; enabled; vendor preset: enabled)
     Active: active (running) since Tue 2026-05-05 01:07:35 UTC; 2 months 26 days ago
   Main PID: 992076 (google_osconfig)
      Tasks: 11 (limit: 19173)
     Memory: 457.8M
     CGroup: /system.slice/google-osconfig-agent.service
             └─992076 /usr/bin/google_osconfig_agent

OSConfigAgent Info: Writing inventory to guest attributes   # every ~10 min

Status: ✅ active (running), inventory reporting on a ~10-minute cadence.

Notes on SSH output

Two warnings appear on every IAP-tunnelled session and are not errors:

To increase the performance of the tunnel, consider installing NumPy — optional throughput optimisation for IAP TCP forwarding.
connection is not using a post-quantum key exchange algorithm — the server's OpenSSH predates PQ key exchange; flags a theoretical "store now, decrypt later" exposure. See https://openssh.com/pq.html.

3. Creating the Scheduled Patch Deployment

The deployment uses a zone-by-zone rollout with a strict disruption budget so the taxi service is never fully offline.

Navigate to the setup

Go to Compute Engine → VM Manager → Patch in the Google Cloud console.
Click + Create patch job.

Target VMs

Setting	Value
Zones	`europe-north1-a`
Filter VM instances	Name prefix → `att-nightly-tms-backend`

This targets both active servers (1b and 2b) and excludes the stopped confidential instances.

Patch configuration

Setting	Value
Patch job name	lowercase with hyphens, e.g. `security-updates-nightly`
APT settings → type of upgrade	`apt-get upgrade`
Exclude packages	(blank)

Why apt-get upgrade: it is the conservative option — it will not remove existing packages or pull in new dependencies, which prevents unexpected system breakage. (apt-get dist-upgrade would do both.)

Scheduling

Setting	Value
Schedule type	Recurring schedule — not One-time, or the automation runs once and stops
Frequency	Weekly
Day	Off-peak, e.g. Saturday
Time	Off-peak, e.g. 00:00
Duration	60 minutes (1 hr) maintenance window

Rollout options — crucial for zero downtime

Setting	Value
Zone rollout	One zone at a time
Zone disruption budget — type	Number of VMs
Zone disruption budget — value	1

Alternative: use Percentage at 50% (or 25% as the fleet grows) instead of a fixed count.

Why this matters: a budget of 1 forces VM Manager to patch and reboot 1b first, wait for the OS Config agent to report a COMPLIANT and healthy status, and only then proceed to 2b. If the first VM fails, the deployment halts — guaranteeing at least one backend server stays up to serve traffic.

Advanced options

Setting	Value
Post-patch reboot option	Default — the agent decides whether a reboot is needed from OS signals
Pre-patch script	(blank)
Post-patch script	(blank)

Then click Deploy.

4. Monitoring and Operations

Once the schedule is active, monitor it natively in the console.

Checking compliance

VM Manager → Patch → Dashboard Review which VMs are up to date and which are missing specific security packages across the project.

Auditing executions

VM Manager → Patch → Jobs run Inspect execution logs, duration, and reboot signals per instance after each scheduled maintenance window.

Suggested routine

Cadence	Action
After each window (Sat 00:00–01:00)	Check Jobs run for failures or halted rollouts
Weekly	Review the Dashboard for lingering non-compliant VMs
On fleet change	Re-check that the `att-nightly-tms-backend` name prefix still matches the intended set, and that new VMs have the agent running

Appendix — Current Fleet State

Instance	Zone	State	OS Config agent
`att-nightly-tms-backend-1b-nightly`	`europe-north1-a`	Running	✅ active since 2026-05-04, COMPLIANT
`att-nightly-tms-backend-2b-nightly`	`europe-north1-a`	Running	✅ active since 2026-05-05, reporting inventory
`tms-confidential-instance-nightly`	—	Stopped	n/a — excluded by name-prefix filter
`tms-confidential-template`	—	Stopped	n/a — excluded by name-prefix filter

Stopped instances are skipped by the patch job. If they are ever started, confirm the agent is running before relying on VM Manager to patch them.