SSL certificate renewals: Current process (2026 — Let's Encrypt + Vault, all wildcards)

Current, unified renewal process for all four Ahola wildcard certificates (attracs.com, aholatransport.com, aholadigital.com, online.aholadigital.com). Supersedes the older DigiCert-era per-cert pages under SSL certificate renewals — those are kept for reference/legacy, not deleted. Mirrors the repo runbook attracs-linux-playbooks/docs/CERTIFICATE-RENEWAL.md.
Last full renewal: 2026-07-15 · next round: start ~2026-09-10.

Certificate Management & Renewal

Runbook for the four Ahola wildcard SSL certificates: *.attracs.com, *.aholatransport.com, *.aholadigital.com, and *.online.aholadigital.com.

Where to find certificates (source of truth)

Certificates are created via Let's Encrypt automation and stored in Ahola Group Vault: https://vault.aholagroup.com/. That is the single source of truth for the current certificate + private key of each domain — look there first, don't hunt for files on individual hosts.

Example path (aholadigital):

https://vault.aholagroup.com/ui/vault/secrets/certs/kv/issuer/aholadigital.com

(holds the fullchain cert.pem + key).

Provider / cadence: the certs are short-lived ~90-day wildcards issued via
ACME (Let's Encrypt-style; the 2026-07 round was issued by ZeroSSL DV) — not
the old annual DigiCert certs. Renewals are frequent — do not let the calendar drift.

Last full renewal: 2026-07-15. Next round: start ~2026-09-10.

Cert	Expiry (next renewal)
`*.attracs.com`	2026-09-17
`*.online.aholadigital.com`	2026-09-19
`*.aholadigital.com`	2026-09-21
`*.aholatransport.com`	2026-10-04

Certificates at a glance

Cert	Where it is deployed
`*.attracs.com`	7 GCP LBs · 4 HAProxy frontends · Windows online2 (RDWeb)
`*.aholatransport.com`	GCP LBs only (stable + nightly) — not on the frontends
`*.aholadigital.com`	10 GCP LBs · 1 Hetzner Cloud LB
`*.online.aholadigital.com`	Windows RDS (2 RD gateways + Broker deployment properties)

Cert file shapes (differ per platform!)

From Vault you get cert.pem (fullchain = leaf + intermediate) and a separate key.

Target	Format needed	How to build
HAProxy frontends	one combined PEM, key + fullchain	`cat certkey.key cert.pem > <name>_YYYY-MM.pem`
GCP load balancers	fullchain + key as separate fields	use `cert.pem` and `certkey.key` directly
Windows / IIS / RDS	PFX (PKCS#12)	`openssl pkcs12 -export -in cert.pem -inkey certkey.key -out <name>_YYYY-MM.pfx`

Naming: name files by issue date, e.g. attracs_2026-07.pem / .pfx.

Verify before deploying (run these yourself — key material stays local):

openssl x509 -in cert.pem -noout -subject -issuer -dates
openssl x509 -in cert.pem -noout -ext subjectAltName        # MUST include DNS:*.<domain>
grep -c "BEGIN CERTIFICATE" cert.pem                         # >=2 (leaf + intermediate)
openssl x509 -in cert.pem -noout -modulus | openssl md5      # must equal:
openssl rsa  -in certkey.key -noout -modulus | openssl md5

`*.attracs.com`

Do these in order.

1. GCP load balancers (7)

attracs-services, attracs-nightly-services, attracs-logistics, attracs-logistics-nightly, eazytms, eazytms-nightly, attracs-devops.

gcloud compute ssl-certificates create attracs-YYYY-MM --project=<PROJECT> --global \
  --certificate=cert.pem --private-key=certkey.key
gcloud compute target-https-proxies describe <proxy> --project=<PROJECT> --global \
  --format="value(sslCertificates.basename())"           # see current list first
gcloud compute target-https-proxies update <proxy> --project=<PROJECT> --global \
  --ssl-certificates=attracs-YYYY-MM,<other-certs-to-keep>

⚠️ --ssl-certificates REPLACES the whole list. attracs-devops,

✅ Safer: use the GCP Console UI for shared / multi-cert LBs. Because the CLI replaces the whole cert list, updating several certs (or a proxy that serves more than one) from gcloud is risky — one omission knocks a domain offline. In the GCP Console (Network services → Load balancing → the LB's frontend → Certificate, or Security → Certificate Manager) you add/swap certs per item without rewriting the list. Prefer the UI whenever a proxy serves multiple certs; use the CLI only when you're certain of the full list.

attracs-logistics, attracs-logistics-nightly, eazytms, eazytms-nightly

also serve *.aholadigital.com — keep that cert in the list or it goes offline.

2. HAProxy frontends (3 via Ansible)

Host	IP	Config
`attracs-frontend-1`	95.216.3.175	`attracs.cfg.j2`
`attracs-frontend-2`	95.216.1.224	`attracs.cfg.j2` (test here first)
`tailorframe-2`	35.228.95.105	`attracs.cfg.j2`

Deploy is by direct scp of the combined key+fullchain PEM — NOT via frontend-files.yml/git (frontend/files/cert/ in the repo is frozen at 2023 and is not the live source).

KEY=~/.ssh/attracs-backend
DEST=/etc/attracs-frontend/cert/attracs_YYYY-MM.pem
scp -i "$KEY" attracs_YYYY-MM.pem root@95.216.1.224:"$DEST"
scp -i "$KEY" attracs_YYYY-MM.pem root@95.216.3.175:"$DEST"
scp -i "$KEY" attracs_YYYY-MM.pem root@35.228.95.105:"$DEST"
# bump the crt path in attracs.cfg.j2, then push CERT-ONLY:
ansible-legacy --vault-password-file=.vault_password_file -i inventory/frontend.ini \
  -l 'attracs-frontend-1,attracs-frontend-2,tailorframe-2' \
  frontend/haproxy/haproxy-update-config.yml

⚠️ Use ansible-legacy (→ ~/ansible-legacy/bin/ansible-playbook, ansible-core
2.14; set up with ./setup-ansible-legacy.sh). Frontends run Ubuntu 16.04 / Python
3.5; modern ansible aborts with "3.6 minimum".
⚠️ Run haproxy/haproxy-update-config.yml directly (cert-only). Do NOT use the

scripts/frontend-update-configs-limit-*.sh wrappers — they run the full

frontend-update.yml (re-templates nginx, updates DNSMadeEasy, posts to Slack).
⚠️ attracs.cfg.j2 no longer references ahola_*.pem — don't re-add it (tailorframe-2
lacks the file and haproxy -c will fail).

3. ☢️ attracs-devops-3 (51.137.59.143, Azure "Online-api2") — HAND-MAINTAINED

Its live /etc/haproxy/haproxy.cfg is hand-maintained (~60 backend_online_api_* backends incl. production /cramo → 10.2.0.10:10504). The repo's internal.cfg.j2 is a stale template that does NOT represent this host.

Running the Ansible frontend play against devops-3 overwrites the live config and
takes down production — it caused a ~40 min CRAMO outage on 2026-07-14. Never run the
playbook against devops-3.

Cert change on devops-3 = edit the live config by hand:

# on attracs-devops-3, as root (scp the new attracs_YYYY-MM.pem to /etc/attracs-frontend/cert/ first):
cd /etc/haproxy
cp haproxy.cfg /root/haproxy.cfg.bak-$(date +%F)
sed -i 's#<old-cert-filename>#attracs_YYYY-MM.pem#' haproxy.cfg    # swap ONLY the bind crt path
haproxy -c -f haproxy.cfg && systemctl reload haproxy

(Ansible's backup: yes also leaves a timestamped /etc/haproxy/haproxy.cfg.<ts>~.)

4. Windows — online2.attracs.com (RDWeb)

RDS env, needs a PFX. Servers: RDS broker 10.2.0.5, RDS gateway 10.2.0.6. Per server: copy .pfx to C:\certholder\ → IIS → Server Certificates → Import → Default Web Site → Edit Bindings → https/443 → select new cert. If IIS binding alone doesn't take, also set it in Server Manager → RDS → Edit Deployment Properties → Certificates.

`*.aholatransport.com`

GCP-LB-only (no longer on the Hetzner frontends). Two projects: * stable — attracs-services-195009 * nightly — att-nightly

Upload fullchain + key and swap on each HTTPS proxy (same gcloud steps as attracs §1). --ssl-certificates replaces the list — include any co-served certs.

`*.aholadigital.com`

Cert source: Ahola Group Vault (.../certs/kv/issuer/aholadigital.com). Scope: 10 GCP LBs + 1 Hetzner Cloud LB (NOT Windows, NOT the HAProxy frontends).

GCP LBs (10): common-servers-01, stable-servers-01, nightly-servers-01, stable-biztalk-01, nightly-biztalk-01 (aholadigital-only); attracs-devops, attracs-logistics, attracs-logistics-nightly-6519, adi-eazytms-stable, adi-eazytms-nightly (shared with *.attracs.com → keep the attracs cert in the --ssl-certificates list on swap). Same gcloud steps as attracs §1.

Hetzner Cloud LB: console project 2477407 → Security → Certificates → Add, then Load Balancers → Services → edit → update cert. Verify: lb-road-nightly.aholadigital.com, integrationservices[-test].aholadigital.com.

`*.online.aholadigital.com`

Windows RDS deployment (GCP RDP env). Needs a PFX. * RDS broker: 10.170.0.64 * RD gateways: rdsgw-01 10.170.0.59, rdsgw-02 10.170.0.60

Import the PFX (IIS/bind 443) on both gateways, then set it in the Broker: Server Manager → Remote Desktop Services → Edit Deployment Properties → Certificates — set for all 4 roles (RD Connection Broker SSO, RD Connection Broker Publishing, RD Web Access, RD Gateway) until each shows Trusted/OK with CN=online.aholadigital.com. Session-host RDP-Tcp listeners (rdshost-0..4) keep their internal per-host certs (auto-enrolled) — leave them, they are correct. Verify end-to-end: launch a RemoteApp via rdgw.online.aholadigital.com and confirm the publisher shows as trusted online.aholadigital.com.

Verification (any platform)

echo | openssl s_client -servername <domain> -connect <ip-or-host>:443 2>/dev/null | openssl x509 -noout -enddate
echo | openssl s_client -servername <domain> -connect 127.0.0.1:443 2>/dev/null | openssl x509 -noout -enddate   # on-host = proves reload
curl -sSI https://online2.attracs.com/RDWeb/ 2>&1 | head -5     # Windows RDWeb: clean 200/301, no SSL error

Expect notAfter = the new cert date. (HAProxy reloads keep the systemd Active: since timestamp pinned to the original start — normal; the cert date is the real proof.)

Incident post-mortem — CRAMO 503 outage (2026-07-14, ~40 min, ~10:21–11:03)

Cause: pushing the attracs.com cert to attracs-devops-3 via the Ansible play

overwrote its hand-maintained /etc/haproxy/haproxy.cfg with the repo's stale

internal.cfg.j2, wiping ~60 backends incl. production /cramo → 503 <NOSRV>.

Fix: restored from Ansible's automatic backup (haproxy.cfg.<ts>~), then applied

the new cert surgically (one-line sed on the bind path) + haproxy -c + reload →

CRAMO back to 201.

Prevention: devops-3 is hand-maintained — never Ansible-deploy to it (see §3);

added a ☢ DANGER header to internal.cfg.j2.

Retired / dropped targets (do NOT update)

Target	Why dropped
`attracs-meshmoon-1` (135.181.3.243)	No longer ours; config hand-edited, never synced from repo
`attracs-devops-2` (172.31.1.100)	Old online-api-1, shut down / unreachable
`wac.attracs.com`	No longer in use
`online.attracs.com` (Citrix), xenapp2 IIS	Decommissioned

Out of scope — managed by others

The apex website domains aholatransport.com and aholadigital.com serve Google-managed certs (Google Trust Services, auto-renewing). These are websites handled outside SecDevOps and are not part of these wildcard renewals — don't try to "fix" them during renewal/verification. (Note: attracs.com's apex is on our ZeroSSL wildcard.) Confirmed 2026-07-15.

Lessons / gotchas

Repo configs drift from live (internal.cfg.j2 = devops-3, meshmoon.cfg.j2) —

diff against the running config before any deploy.

Certs are ~90-day — track the next-renewal dates above; start ~Sep 10 2026.
Frontend cert deploy = direct scp, ansible-legacy only, cert-only play.
The {% if haproxy_certificates %} block in the configs is not dead code — the

meshmoon play's find task discovers LE certs and appends them; keep it.

References

ClickUp SSL cert docs (doc 2c5q6-69975, DigiCert-era / being updated for LE+Vault):

list 2c5q6-323295 · attracs 2c5q6-11726 · aholatransport 2c5q6-216508 ·

aholadigital 2c5q6-261195 · online.aholadigital 2c5q6-261935 · Hetzner 2c5q6-10823.

Vault: https://vault.aholagroup.com/