25.10.2023 - TMS Stable machine type update

Machine type update was scheduled for TMS stable backends attracs.tms.backend.3 and attracs.tms.backend.4 * Monday 23.10.2023 * Noticed that the instances are in a Managed Instance Group (MIG), backed off update until it was confirmed that instance disks don't differ, like they do in the MIG instances devops.4 and devops.5 * Tuesday 24.10.2023 * Confirmed that disks are ok, proceeded to create a new instance template with the instance template copy -function so that all settings are preserved from the current instance template, only the machine type is changed. * Updated the instance template for the MIG and started a restart update. * Noticed that the instances wouldn't get restarted, so the new instance template wouldn't be applied. * Tried restarting backend 4 manually, instance template still didn't change. * Backend 4's assigned IP address would be lost after each restart but this was easily reassigned back. * Due to these problems with the update, the update was postponed another day. * Wednesday 25.10.2023 * Final attempt was tried, now using the replace function instead of the restart. Instances still didn't get replaced * Attempted to manually delete backend 4 to force instance template update * After doing this and waiting for about four minutes, backend 4 was being re-created by Google but now backend 3 started getting deleted by Google * Both instances got replaced and the http-server network tag was lost * Ingress traffic to the applications, including Google's health check probes started failing * Since probes couldn't connect, health checks started failing * Since health checks failed, the Load Balancer couldn't decide which backend to route traffic to, since both backends showed unhealthy. Load Balancer error was failed_to_pick_backend * We decided to re-route traffic from Google LB's to our HAProxies in Hetzner. DNS records were changed to point to HAProxies and HAProxies were configured. * Applications started working again but now they were not protected by Google Cloud Armor that we use to avoid DDoS and CVE vulnerability attacks.

We realized the root cause was a firewall rule that allowed Google health check probes only to instances that had the network tag http-server. We decided to create a new firewall rule that allows traffic from all TCP ports to all instances in the network from Google's health check probe IP address ranges. All of our health checks came back alive after this change.

We decided to create new backends attracs.tms.backend.5 and attracs.tms.backend.6 that were not under any MIG, instead opting for Unmanaged Instance Groups (UIG) that do not use instance templates. We also came to the conclusion that our infrastructure currently doesn't support us using MIGs. By using UIGs we could still use Google Cloud Armor and not have to worry about the instance templates failing.

We will probably phase out our MIG in favor of the UIGs, this is still a to do.

One issue with the UIGs is that they only support a single-zone inside of a region, unlike MIG that is divided between multiple zones inside of a region. Google does not guarantee that a single zone will stay up 100% of the time but they do promise that a region is always available. With this setup all of our backends would be under a single zone. At the time of writing this page we are testing if we can have use multiple UIGs so we could have backends in two different zones.

Still to do is to switch our backends back to Google Load Balancer from HAProxy.