Skip to content

Vault (Hashicorp)

Used by Ahola Digital to manage secrets for applications.

Vault in general

Vault secret engine = Directory, contains secrets, could be for example kv-v2 or kv-v2-nightly. Vault secret = File inside of a secret engine. Vault kv/key-value = key : value pairs inside of a secret

New Vault

Platform Hetzner Cloud
Project Digital-Devops
Instance hetzner-services-stable-1
also HA'd to hetzner-services-stable-2
Repository https://github.com/Attracs/docker-hashicorp-vault/tree/main/hetzner-vault
Dockerized? Yes
Address vault-hetzner.aholadigital.com
Internal address ?

Old Vault

Platform GCP
Project attracs-devops
Instance devops.4 (devops-instance-group-fng2)
also HA'd to devops.5 (devops-instance-group-x3wj)
Repository
Dockerized? Yes
Address vault.attracs.com
Internal address vault-internal.attracs.com

Old Vault

  1. We manually write vault .yml.ctmpl configuration files attracs.yml.ctmpl (stable) and attracs-nightly.yml.ctmpl under attracs-application-configs/application-configs/application-name/
  2. We copy these files with an Ansible script to chosen (limiter limited) servers inside /etc/vault-agent.d/templates/application-name.
  3. We authenticate instances to Vault based on which roles and service account permissions they have.
    1. Roles are: 
      vault list auth/gcp/roles
Keys
city-servers
devops-servers
hetzner-servers
meshmoon-servers
nightly-servers
production-servers
road-servers
tailorframe-servers
tms-production-servers
  4. Then, using the vault-agent configuration in the server in /etc/vault-agent/agent_config.clf we run through if and for-loops and write the key-values from Vault to another file in the directory/etc/application-name/ to files attracs.yml and/or attracs-nightly.yml. These files contain the actual Vault secrets in plain text.
  5. Then the application reads the .yml files.

How vault "gce" connection works

devops-compute-engine service account is given the following permissions to attracs-vault@attracs-devops.iam.gserviceaccount.com service account via a custom role called vault-service-account-role:

compute.instances.get
iam.serviceAccounts.get
compute.instanceGroups.list
iam.serviceAccountKeys.get
resourcemanager.projects.get

How to find the status of Vault

$ vault status -address https://vault-internal.attracs.com It should print you a similar output:

Key                      Value
---                      /-----
Recovery Seal Type       shamir
Initialized              true
Sealed                   false
Total Recovery Shares    5
Threshold                3
Version                  1.9.3
Build Date               n/a
Storage Type             gcs
Cluster Name             vault-cluster-3fda5784
Cluster ID               28d896c7-db9a-9d6f-7b19-ffd20a12c55a
HA Enabled               true
HA Cluster               https://vault.attracs.com:444
HA Mode                  active
Active Since             2023-08-17T08:06:14.527262972Z

How to use the Vault for an application

TODO

The now (implemented at least 50%)

We use the new vault by having applications fetch secrets from vault and saving those to memory. Permissions are managed with GCP service accounts by giving a list of service accounts to a vault role that determines what secrets instances are allowed to read based on the service account they are running under.

Troubleshooting

ModuleNotFound error when trying to install vault from ansible-playbook

Trying to install vault from ansible-playbook docker-vault gives me the following ModuleNotFound error:

TASK [Stop vault instance] ***********************************************************************************************************************************************************************************************************
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: ModuleNotFoundError: No module named 'docker'
failed: [adi-devops-confidential-instance-stable-2] (item=docker-vault) => {"ansible_loop_var": "item", "changed": false, "item": "docker-vault", "msg": "Failed to import the required Python library (Docker SDK for Python: docker>=5.0.0 (Python >= 3.6) or docker<5.0.0 (Python 2.7)) on adi-devops-confidential-instance-stable-2's Python /usr/bin/python3. Please read the module documentation and install it in the appropriate location. If the required library is installed, but Ansible is using the wrong Python interpreter, please consult the documentation on ansible_python_interpreter, for example via `pip install docker` (Python >= 3.6) or `pip install docker==4.4.4` (Python 2.7). The error was: No module named 'docker'"}

Solution: install docker-compose to the target server: (todo, fix or make this a part of the server bootstrap)

sudo apt install docker-compose

Image not found error when trying to install vault from ansible-playbook

failed: [adi-devops-confidential-instance-stable-1] (item=docker-vault) => {"ansible_loop_var": "item", "changed": false, "errors": [], "item": "docker-vault", "module_stderr": "", "module_stdout": "", "msg": "Error starting project 404 Client Error for http+docker://localhost/v1.47/images/create?tag=latest&fromImage=vault: Not Found (\"manifest for vault:latest not found: manifest unknown: manifest unknown\")"}

Solution: vault docker image has been deprecated. Use hashicorp/vault instead.

Cannot get vault status, error 403 forbidden

URL: GET https://vault-internal.attracs.com/v1/sys/seal-status
Code: 403.

You are not in an allowed network

Resultant ACL check failed

Likely one of the vaults are sealed. You can test this from the GUI-level by "spamming" a vault command in the Vault CLI:

vault list auth/gcp/roles

You will receive "vault is sealed" error intermittently if this is the case.

Figuring out the faulty Vault instance

To find out which vault you will have to restart, go to the servers hosting vault and check the IP address with docker inspect -f '{{ .NetworkSettings.Ports }}' hetzner-vault

You can then try to call the status of the vault instance like so:

root@hetzner-services-stable-2:/opt/hashicorp_vaults/vault_stable/configuration# vault status -address http://10.0.1.6:8200
Error checking seal status: Error making API request.

URL: GET http://10.0.1.6:8200/v1/sys/seal-status
Code: 503. Errors:

* Vault is sealed

If the result is "Vault is sealed" proceed to restart the docker container using the docker-compose file. You can find the location of the docker-compose from the inspect as well: docker inspect -f '{{ index .Config.Labels "com.docker.compose.project.working_dir" }}' hetzner-vault

Subpages