Troubleshooting Confidential Computing environment

Response did not return ClientToken, client token not set

When restarting an application in Managevisor, the application goes into a restart-backoff loop and the fatals with this error:

attracs-golang@v0.0.0-20240311162622-d4513dfd8512/util/mlogger/logger.go:724 05:23.55 FATAL: response did not return ClientToken, client token not set

Also running this command results in an error: config-diff-vault

10:42:36 9.4.2024 07:42.36 [config.vault] Loading from Vault server 07:42.36 ERROR: response did not return ClientToken, client token not set

This can be debugged from the confidential instances like so: 1. Check that the default service account is correct, it should be running the confidential SA: 1. gcloud config list

[core]
account = confidential@<gcp-project>.iam.gserviceaccount.com
disable_usage_reporting = True
project = <gcp-project>
Your active configuration is: [default]

2. Try getting a client token from Google: 1. The audience in the end needs to be set to something, but it shouldn't matter what it is.

curl -iH "Metadata-Flavor:Google" "http://metadata/computeMetadata/v1/instance/service-accounts/default/identity?format=full&audience=abc123

Response:

HTTP/1.1 200 OK
Metadata-Flavor: Google
Date: Tue, 09 Apr 2024 08:00:35 GMT
Content-Type: text/html
Server: Metadata Server for VM
Content-Length: 1230
X-XSS-Protection: 0
X-Frame-Options: SAMEORIGIN
eyJ...

3. If this all works, it means that GCP should be fine. You should move to debugging Hetzner Vault. Try logging into the stable and nightly Vault at https://vault-hetzner.aholadigital.com and https://vaullt-hetzner-nightly.aholadigital.com and see if they differ in any way from each other: 1. If you get "Error Authentication failed: TypeError: Network request failed", some of the Vaults may be down on the affected environment. You should check the Hetzner Cloud servers and the docker containers to see if there is anything weird with them.