Skip to content

Status TV Dashboards

The TVs don't show the team's normal Grafana dashboards directly. They show TV versions of them: dashboards that fit exactly on one 2560×1440 screen, because a TV in kiosk mode can't scroll and the originals were 40–110 grid units tall (only about 37 fit). The TV versions live in the Grafana folder Status-TV and are generated by a script in the devops-status-tv repo.

  • Every panel is a copy of a panel in an original dashboard. The originals are never changed, and their owners can keep editing them.
  • Each TV dashboard fits in 36 grid units (one screen, no scrolling) and hides the variable and time-picker bar.
  • Each has a one-line description at the top, so people walking past know what they are looking at.
  • Collapsed rows of the originals (e.g. Hiddenalerts, RPC / PubSub, Cloud SQL insights) are left out. The TVs never showed them, because kiosk mode can't open a row.
  • Exception: Road 7 is built from scratch, not copied (see Road 7 and Road Web Infra apps).

What's on the screens

Both playlists change dashboard every 2 minutes. A full round takes 12 minutes on CITY (6 dashboards) and 14 minutes on ROAD (7 dashboards).

Left TV – playlist Status TV - CITY (hetzv9p)

# Dashboard (uid) Built from Shows
1 City 1 · TMS overview (stv-city-1) TMS stability monitor Taxi-grid API traffic, Drivers app versions, PostgreSQL status/connections/idle sessions, TMS server goroutines, CPU, memory
2 City 2 · TMS services & WebSocket (stv-city-2) TMS stability monitor, TMS taxi grids Load balancer, memory per service, TMS status codes, Memcache, DB query results, Planner/Drivers/customer WebSockets
3 City 3 · Planner automations (stv-city-3) TMS stability monitor Planner offer automation per organisation and per server
4 City 4 · Taxi grids (stv-city-4) TMS taxi grids Queue totals, info-query rate/errors/latency, vehicles and trips per area
5 City 5 · Production Vault (stv-city-5) Production Vault Disk, CPU and memory of the 3 Vault nodes (to be replaced, see Planned)
6 City 6 · Security updates (stable) (stv-city-6) Google Cloud Monitoring / Security updates / Nightly / Stable Stable servers that need a reboot, pending security and other apt updates

Right TV – playlist Status TV - ROAD (hetrgxv)

# Dashboard (uid) Built from Shows
1 Road 1 · eazytms Cloud SQL & VM (stv-road-1) eazytms Cloud SQL latencies, rows, transactions, CPU, disk; eazytms VM goroutines, memory, CPU
2 Road 2 · eazytms services (stv-road-2) eazytms HTTP, RPC, Pub/Sub, EazyTMS status, Attracs Online REST/Pub/Sub, WebSocket, map services
3 Road 3 · Ahola Transport (stv-road-3) Service Desk Ahola The whole Ahola Transport section: services, admin queues, RDS hosts, Biztalk, MSSQL, backend server
4 Road 4 · Ahola Special & logs (stv-road-4) Service Desk Ahola Ahola Special services, AO load balancer 5XX, AO log exception/perf maps, DB backup log
5 Road 5 · Road Web Infra (stv-road-5) Road Web Infra Cloud SQL, PostgreSQL connections, road-confidential VM CPU and memory
6 Road 6 · Backendsrv-1 (stv-road-6) Backendsrv-1 Agents, CPU and memory (gauges + Zabbix), AO server CPU per process, Windows process working set / private bytes, network, C: disk
7 Road 7 · Road Web Infra apps (stv-road-7) built from scratch (supervisord metrics) Every supervisord program on road-confidential-instance-stable-1/2: a tile per program with state and uptime, Fatal / Stopped / Restarts counts, a "Not running" table

Small exceptions to "copied as-is": on City 6 the tables hide the Time, env, instance and job columns (they pushed the values off the screen), and the environment is fixed to stable. On City 2 the untitled "Panel Title" panel from TMS taxi grids is called "Drivers WebSocket messages".

Road 7 and Road Web Infra apps (supervisord)

The Road Web Infra servers run 40–50 apps each under supervisord. Two dashboards show whether they are running:

Dashboard Where Servers
Road 7 · Road Web Infra apps (stv-road-7) Status-TV folder, ROAD playlist stable-1/2, fixed
Road Web Infra apps (road-web-infra-apps) ROAD folder, for desk use Environment (stable / nightly) and Server variables at the top left

Colours of the program tiles

Tile Meaning
green + uptime RUNNING
light orange + uptime RUNNING, but restarted within the last hour
light orange "STARTING" being started (a program that is often STARTING is restart-looping)
red "FATAL" (or BACKOFF / EXITED / UNKNOWN) crashed and supervisord gave up, or keeps failing. Counted in the Fatal box
grey "STOPPED" stopped by hand or autostart=false. Counted in Stopped on purpose

Where the data comes from: supervisord-metrics.py runs every minute on each Road server (systemd timer). It reads supervisord's own API over its unix socket and writes supervisord.prom for node_exporter's textfile collector, so Prometheus scrapes it with the other host metrics. It only reads; it never starts or stops programs. Script, install playbook and metric list: attracs-linux-playbooks supervisord/metrics/. Installed on road-confidential-instance-stable-1/2 and -nightly-1/2.

Changing these dashboards: both are generated by grafana/build.py in devops-status-tv from one shared definition (apps_panels()), so they always look the same.

  • Road 7: build and push like any TV dashboard (see below).
  • Road Web Infra apps: the editor service account can't write to the ROAD folder, so it's imported by hand: Grafana → Dashboards → New → Import → grafana/dashboards/road-web-infra-apps.json, folder ROAD, overwrite the existing one.
  • The Server list comes from the host / instance labels of the host_apt_pending_updates metric (apt check), so a new Road server appears once it has both the apt check and the supervisord script.

Who can do what in Grafana

Service account Role Used by
indola-status-tv (login sa-1-indola-status-tv) Viewer The TVs (token on the Pi, see Status TV)
ai-dashboard-editor (login sa-1-ai-dashboard-editor) Viewer everywhere, Editor only in the Status-TV folder The scripts below (token as grafana_editor_token in your local .env, value in Proton Pass). Never deployed to the Pi

A source dashboard must be visible to both accounts: the editor to copy panels from it, and the TV account to show the data. If a dashboard is in a restricted folder, give both accounts Viewer on that folder (folder → Manage permissions).

Playlists are edited by people in the Grafana UI. The editor account can't change them.

Change a TV dashboard

Don't edit the TV dashboards in the Grafana UI: the next upload overwrites the change. Edit the layout in the repo instead.

cd devops-status-tv                    # needs grafana_editor_token in .env
$EDITOR grafana/build.py               # layouts (P(...) lines) and descriptions (DESC)
python3 grafana/build.py               # fetches the source dashboards, writes grafana/dashboards/*.json
python3 grafana/proxy.py &             # local token proxy for screenshots (Ctrl-C / kill when done)
python3 grafana/shot.py stv-city-2     # stv-city-2.png at TV resolution – look at it
python3 grafana/push.py stv-city-2     # upload (omit the uid to upload all)
git commit -am "TV: ..." && git push
  • build.py stops with an error if a dashboard is taller than 36 units or panels overlap.
  • A layout line is P(source, panel_id, x, y, w, h) on Grafana's 24-column grid. Panel ids are in the source dashboard's JSON (Share → Export → JSON, or grafana/.cache/<uid>.json after a build).
  • Because build.py fetches the originals every time, changes to an original's panels (queries, thresholds) flow into the TV copy on the next build and push. If a panel is deleted from an original, the build fails with a KeyError and its line needs updating.
  • Grafana keeps every uploaded version (dashboard → Settings → Versions), so a bad upload can be rolled back there.
  • Edits to a dashboard show up on its next turn in the playlist. A restart is only needed when the playlist itself changes.

Add a dashboard to a TV

  1. Make sure both service accounts can view the source dashboard (see above).
  2. In grafana/build.py: add the source uid, a dash("stv-city-7", ...) block with the panels, and a DESC line. Use uids stv-city-N or stv-road-N.
  3. Build, screenshot, push as above.
  4. Grafana → Dashboards → Playlists → add it to Status TV - CITY or Status TV - ROAD, save, and restart the screens: ssh statusadmin@10.10.75.9 '~/status-tv/restart.sh'.

Screenshots without the TVs

Grafana has no image renderer plugin, so shot.py runs headless Chrome locally against proxy.py, which adds the editor token the same way nginx does on the Pi. Notes:

  • Wait time defaults to 25 s. Dashboards with many Zabbix panels (Road 3) need more: python3 grafana/shot.py stv-road-3 out.png 40.
  • A fourth argument replaces ?kiosk with your own URL parameters, e.g. a normal (non-kiosk) view with variables: python3 grafana/shot.py road-web-infra-apps out.png 25 "?orgId=1&var-env=nightly" (the dashboard must be readable by the editor account).
  • "No data" in a screenshot is real if the same panel is empty in the original dashboard too.

To see what the TVs actually show, use Take a screenshot on the Pi.

Known gaps

  • Some copied panels show No data, the same as in the originals: on Road 2 HTTP, Memcached, Hit rate, Read rate; on Road 3 Ahola Ferry file count, Ahola Ebookings, Ahola Live Server Status Board, Web Applications Status. Their queries need fixing in the original dashboards.

Planned

Vault health screen (replaces City 5). The current Vault dashboard only has OS metrics from Zabbix. On 2026-10-13 the Zabbix template HashiCorp Vault by HTTP was linked to the three Vault hosts. When it is collecting, City 5 becomes a Vault health screen with:

  • sealed / unsealed and active / standby per node (large status tiles)
  • cluster (Raft) health and leader
  • request rate and latency, audit log failures (must stay 0)
  • lease and token counts, disk per node

Status 2026-10-15: host macros {$VAULT.HOST} (Tailscale IP), {$VAULT.API.SCHEME} (https) and {$VAULT.API.PORT} (8200) are set, but the Zabbix server gets no answer from https://100.x.x.x:8200/v1/sys/health (Tailscale ACL or Vault listener address to be checked). A read-only Vault token for Zabbix ({$VAULT.TOKEN}) and the telemetry block in the Vault config are still missing. ClickUp: Add vault-aholagroup cluster (Prod/nightly) monitoring with alerts.

Before the template was re-linked, deb-x86-hel1-1 had stale Vault items and two firing triggers ("Vault server is not responding", "Token [null] lookup error") left over from an earlier link. Check in Zabbix that they are gone or collecting again.