Skip to content

SCRUM-231 — CI validation of the observability configs

Plan ref: DSH-A5 (docs/11-admin-plane-plan.md). Stacked on SCRUM-230.

What exists

Piece Notes
ci/scripts/observability-checks.sh reads the Prometheus, Loki and Alloy image references from deploy/compose.yaml (validator = deployed version) and runs, in those images: promtool check config, alloy fmt (fails if the output differs from the file, so unformatted files fail too), loki -verify-config; stops at the first failure with the tool's own error
ci/services/Jenkinsfile stage "Observability configs", dashboard job only, before Test, on the agent (the go container has no docker CLI)
ci/dispatcher/Jenkinsfile a change under deploy/observability/ dispatches otomo-dashboard (and only it) — there is no Jenkins job for deploy/
ci/scripts/test-observability-checks.sh stub docker: asserts the images are compose's, a failing promtool fails, an alloy fmt diff fails
deploy/observability/config.alloy reformatted with alloy fmt (tabs): the new check caught that the committed file was not in canonical form

How to verify

sh ci/scripts/test-observability-checks.sh
sh ci/scripts/observability-checks.sh          # on a host with docker

Results at time of writing

  • Stub test: pass.
  • The real script driven through a shim that maps each docker run to the local promtool 3.15 / alloy v1.20 / loki 3.7.8 binaries: all three OK; with a broken scrape_interval appended to prometheus.yml it exits 1.
  • Groovy was re-read, not executed (no Jenkins locally). The Jenkins webhook is currently unreachable (see HANDOVER), so this has not run on the real server yet.

How it was built

DeepSeek run scoped (Landlock) to ci (110 s, ~19k output tokens, reasoning effort low); it reported the unformatted alloy file. Claude reformatted it and ran the script against the real validators, including the failing case.