Skip to content

Deployment and operations

Otomo is deployed as a single Docker Compose stack on one host. This page describes how that stack is laid out, how images are built, tagged and rolled back, how changes are tested and shipped today, what is backed up, and how to see inside a running system.

To install otomo for the first time, follow Deploy otomo on a VM. This page explains what that deployment is made of.

The stack

Everything is described in deploy/compose.yaml, project name otomo. Read its header comments first: they carry the decisions below at the place where someone would change them.

One stack per host

The Compose project assumes it owns the host: one Postgres, one Valkey, one registry, one set of secrets. A second host would be a second, independent stack. Spreading one stack over several hosts (a Swarm overlay with Swarm secrets) is future work and is deliberately a change to compose.yaml only, because services address each other by Compose DNS name (postgres:5432, auth:8080), never by an address.

Networks and published ports

Network Who is on it Purpose
otomo-net Every service The private network; nothing here publishes a port
otomo-edge gateway, edge, edge-proxy, cloudflared, wiki-docs, wiki-site The public tier. An internet-facing container on it cannot reach Postgres or an internal service

Both networks have fixed subnets because some services trust an address: Auth believes X-Forwarded-For only from the gateway's fixed address, and the gateway believes it only from the edges' fixed addresses.

Exactly these ports reach the host:

Port Service Bound to Notes
TCP 8080 gateway 127.0.0.1 by default (GATEWAY_BIND_ADDR, GATEWAY_PORT) The player API. In a normal deployment the edge fronts it
TCP 8090 gateway_dev 127.0.0.1 only The staff plane. Never exposed
TCP 5000 registry 127.0.0.1 only Image registry
TCP 80 and 443 edge All interfaces, profile edge TLS with the host's certbot certificate
UDP 27000 gameplay-proxy All interfaces (GAMEPLAY_PROXY_PORT) Game traffic

The binding is the real control. A port published by Docker bypasses the host firewall's input chain, so a firewall rule documents intent but does not enforce it.

Profiles: what runs when

Profile Adds Use when
(none) Postgres, Valkey, registry, observability, every service Always
edge edge (80 and 443) and wiki-docs The host owns the public name and ports
tunnel edge-proxy and cloudflared A TLS-terminating front (Cloudflare Tunnel, Tailscale Funnel) owns the public name; no host port is published. The front cannot carry UDP, so game traffic keeps its own path
gameservers gs-1, gs-2 You have copied a Godot server export into deploy/gameserver/build/
site wiki-site You serve a game's own website

COMPOSE_PROFILES cannot name both edge and tunnel. See deploy/docs/EDGE-FRONTS.md.

Start-up order and migrations

Each service that owns a database has a one-shot <name>-migrate container (restart: "no") that runs the migrate command and exits; the service itself depends on it completing. up.sh also converges Postgres roles on an existing host before any migration (provision-upgrade.sh), because the provisioning script runs only on an empty data directory.

Data and secrets

Where What Notes
Volume postgres-data Every database One cluster; one database and role per service. max_connections=80 bounds the sum of service pools
Volume blobs Config's content-addressed documents and packs Config writes; Patch mounts it read-only
Volume recordings Replays and other recorded content Patch owns it outright
Volume valkey-data Session and queue state Not durable state; losing it logs players out
deploy/.env Settings and every password Generated; never committed
deploy/secrets/ Signing keys, the TOTP key, the root password, the device pepper, service keys, the player JWKS Generated; never committed; back them up

Passwords arrive through .env and are therefore visible to anyone who can run docker inspect on the host. That is the accepted shape for a single host.

Images, tags and the registry

Every service is a 127.0.0.1:5000/otomo-<service>:<tag> image, built from its own folder (the Go services on a distroless, non-root base; the edge and admin UI on nginx). The tag is the variable OTOMO_<SERVICE>_TAG in deploy/.env, local by default.

Script Does
deploy/scripts/generate-secrets.sh Creates .env, the Valkey config and the secrets (adds missing keys, never overwrites existing ones)
deploy/scripts/up.sh docker compose up plus what YAML cannot say: provision roles, migrate, generate the auth signing key once, probe readiness
deploy/scripts/release.sh <service> [tag] Gated release: pull or build the image, dump the service database when migrations are pending, run its migration before the new code, recreate the service, probe /readyz of it and its dependants, smoke test, append to deploy/releases.jsonl; rolls back automatically when that is safe (deploy/docs/RELEASE.md)
deploy/scripts/rollback.sh --service <svc> --to <tag|release_id> Roll a service back by tag; refused past a migration marked -- otomo:compat breaking (deploy/docs/ROLLBACK.md)
deploy/scripts/tunnel.sh Open the SSH forward to the staff plane
deploy/scripts/backup.sh Write one encrypted backup archive and copy it off the host
deploy/scripts/restore.sh Decrypt, verify and restore an archive into scratch or staging (deploy/docs/RESTORE-RUNBOOK.md)
deploy/scripts/check-compose.sh The CI check of the rendered Compose file's invariants

Every script that runs Compose needs a build profile (OTOMO_PROFILE=dev|staging|production, or /etc/otomo/profile written once per host by deploy/scripts/set-profile.sh); the profile supplies the public host and the channels, and nothing defaults (deploy/docs/PROFILES.md).

Rollback is by tag, not a rebuild. Tags are immutable and nothing prunes the registry, so rollback.sh re-points a service at an earlier tag. Every migration declares on its first line whether the previous release still runs on the schema it leaves (-- otomo:compat backward or breaking); past a breaking one, a rollback needs the pre-release dump release.sh took.

The distroless images have no shell, so there are no Compose healthchecks for them. Readiness is polled from outside against each service's :9090/readyz.

How changes are tested and shipped

Where What When
GitHub Actions Services CI (gateways, gatepolicy, admin UI, edge, and the Compose invariants), Backend services CI (gofmt, vet, -race and tests for each service against throwaway Postgres and Valkey), Matchmaker stack (Session, Allocator, Matchmaker and Gateway built and run together), Game boundary (a game's vocabulary must not reach the platform), Hygiene (rejects tool-generated footers and co-author trailers in PR text and commits) On pushes and pull requests that touch the matching paths
Jenkins An otomo-dispatcher job receives the push webhook for staging, works out which services changed and starts the matching otomo-<service> job. Each runs tests, a smoke test, an image build and the pre-deploy levels (sanity, functional, integration, security, scaling) in a per-build Compose stack. A separate otomo-wiki-docs job rebuilds this documentation After a merge into staging

Branches: pull requests go to staging; main is the release branch the quick start deploys from. Jenkins tests and builds but does not deploy services; someone updates the host by hand with up.sh or release.sh. Only the technical wiki redeploys itself: a push to the docs branch that touches docs/ or services/wiki/ runs deploy/docs/deploy-docs.sh, which builds with mkdocs --strict (a broken link or a nav page that does not exist fails the build) and replaces the running site only if the build succeeds.

Channels are not environments

The release channels dev, staging and live are streams of content inside one stack. They are not separate hosts. Likewise AUTH_ENV (dev, staging, production) only gates development-only features such as the fixture login. How many hosts a team runs, and which branch each follows, is that team's choice; this repository describes one stack.

Backups

deploy/scripts/backup.sh writes one archive: a pg_dump -Fc of every database (discovered from the cluster, not listed in the script), the roles, deploy/secrets, deploy/.env and a manifest naming the git commit, the image tags and the images running. It runs while the stack is up and is encrypted (age or gpg to a public key whose private half stays offline). A systemd timer (deploy/systemd/otomo-backup.timer) runs it daily; a failure alerts through the journal, a Prometheus rule and an optional webhook.

  • Retention: 7 daily and 4 weekly archives, in ~/otomo-backups/ by default.
  • Sensitive: an archive holds the private keys and every password. It is written 0600 in a 0700 directory and should be treated like deploy/secrets.
  • Off-host copy: OTOMO_BACKUP_OFFSITE (rclone:, rsync:, dir: or an explicit none) in /etc/otomo-backup.conf; it has no default because a destination is a policy decision.
  • Tested restore: deploy/scripts/restore.sh restores and verifies an archive, and deploy/scripts/test-backup-restore.sh runs the whole round trip against a local Postgres.
  • Not covered: the registry (rebuildable from the commit in the manifest), Valkey, the 72-hour observability history, and point-in-time recovery. Archives are nightly logical dumps, so the newest may be up to 24 hours old.

A lost device pepper strands every device login, and a lost signing key invalidates every token; both are in deploy/secrets, which is why it is in the archive. Details: deploy/docs/BACKUP.md; the step-by-step operator guide is Back up and restore otomo.

Observability

Metrics and logs are collected inside the stack and reach staff only through the Dashboard.

Piece Job Retention
Prometheus Scrapes every service's :9090/metrics every 15 s, plus node_exporter (host), cadvisor (containers) and postgres_exporter 7 days or 2 GB
Loki Log store on the local filesystem 72 hours
Alloy Tails the stack's container logs, keeps only service and level as labels, pushes to Loki not applicable
Dashboard Runs only named query templates against Prometheus and Loki; the browser never reaches either. Serves the overview, per-service series, log search and the SSE log tail, and the merged audit not applicable

Metric and log names follow one convention: <service>_<thing>_<unit> metrics, JSON log lines with a request_id. To trace a failure, copy the request_id from an error body into the admin website's Logs page.

Alert rules live in deploy/observability/alerts.yml and show on the Overview page as firing alerts. They include PlayerServiceDown, NoFreeGameServers, GameServerPoolEmpty, AccountCreationSpike, RefreshTokenReuseDetected, SigningKeyRotationUnfinished, the gate alerts in Live ops, and MatchmakerDown, MatchmakerQueueStalled and MatchmakerNoCapacity.

Prometheus and Loki publish no port. For debugging, run a query inside the container (docker compose exec prometheus wget -qO- 'http://localhost:9090/api/v1/query?query=up') or start a throwaway socat container on loopback and forward it over SSH (deploy/README.md has both).

Alloy mounts the Docker socket

Log discovery reads /var/run/docker.sock, which is root-equivalent on the host. It is accepted because Alloy publishes no port and runs inside the same trust boundary as the deploy account.

Admin access

The staff plane is reachable only through an SSH tunnel to the host's loopback address:

OTOMO_SSH=you@play.example.com deploy/scripts/tunnel.sh   # then open http://localhost:8090/admin/

One console can also operate a second environment, for example the test copy of the game on another host: gateway_dev relays /env/<name>/... to that environment's gateway_dev over an SSH forward (deploy/environments.example.yaml, GATEWAY_DEV_ENVIRONMENTS_FILE). The registry is optional and absent by default; with it unset nothing changes. The relay opens no new port: the forward is the existing tunnel's, listening on the host's loopback. Every change in a production environment asks for the word production in the console. See deploy/docs/ADMIN-ACCESS.md ("Operating more than one environment").

Use Chrome or Firefox: Safari refuses the Secure cookie on http://localhost. The first sign-in uses the root password in deploy/secrets/admin_auth/root_password to create personal accounts (deploy/scripts/rotate-root-password.sh rotates it). Step by step: Use the admin website and deploy/docs/ADMIN-ACCESS.md.

Sizing and known constraints

Measured on the original 2-vCPU, 3.8 GB host, the stack idles at a few hundred MB. CPU is the real constraint, not memory: builds and go test -race on the same host saturate both cores, so do not release during a busy match. Docker build caches grow with every build and nothing prunes them. Postgres is almost the whole memory footprint.