Deployment and operations¶
Otomo is deployed as a single Docker Compose stack on one host. This page describes how that stack is laid out, how images are built, tagged and rolled back, how changes are tested and shipped today, what is backed up, and how to see inside a running system.
To install otomo for the first time, follow Deploy otomo on a VM. This page explains what that deployment is made of.
The stack¶
Everything is described in deploy/compose.yaml, project name otomo. Read its header
comments first: they carry the decisions below at the place where someone would change them.
One stack per host¶
The Compose project assumes it owns the host: one Postgres, one Valkey, one registry, one set of
secrets. A second host would be a second, independent stack. Spreading one stack over several
hosts (a Swarm overlay with Swarm secrets) is future work and is deliberately a change to
compose.yaml only, because services address each other by Compose DNS name
(postgres:5432, auth:8080), never by an address.
Networks and published ports¶
| Network | Who is on it | Purpose |
|---|---|---|
otomo-net |
Every service | The private network; nothing here publishes a port |
otomo-edge |
gateway, edge, edge-proxy, cloudflared, wiki-docs, wiki-site |
The public tier. An internet-facing container on it cannot reach Postgres or an internal service |
Both networks have fixed subnets because some services trust an address: Auth believes
X-Forwarded-For only from the gateway's fixed address, and the gateway believes it only from the
edges' fixed addresses.
Exactly these ports reach the host:
| Port | Service | Bound to | Notes |
|---|---|---|---|
| TCP 8080 | gateway |
127.0.0.1 by default (GATEWAY_BIND_ADDR, GATEWAY_PORT) |
The player API. In a normal deployment the edge fronts it |
| TCP 8090 | gateway_dev |
127.0.0.1 only |
The staff plane. Never exposed |
| TCP 5000 | registry |
127.0.0.1 only |
Image registry |
| TCP 80 and 443 | edge |
All interfaces, profile edge |
TLS with the host's certbot certificate |
| UDP 27000 | gameplay-proxy |
All interfaces (GAMEPLAY_PROXY_PORT) |
Game traffic |
The binding is the real control. A port published by Docker bypasses the host firewall's input chain, so a firewall rule documents intent but does not enforce it.
Profiles: what runs when¶
| Profile | Adds | Use when |
|---|---|---|
| (none) | Postgres, Valkey, registry, observability, every service | Always |
edge |
edge (80 and 443) and wiki-docs |
The host owns the public name and ports |
tunnel |
edge-proxy and cloudflared |
A TLS-terminating front (Cloudflare Tunnel, Tailscale Funnel) owns the public name; no host port is published. The front cannot carry UDP, so game traffic keeps its own path |
gameservers |
gs-1, gs-2 |
You have copied a Godot server export into deploy/gameserver/build/ |
site |
wiki-site |
You serve a game's own website |
COMPOSE_PROFILES cannot name both edge and tunnel. See deploy/docs/EDGE-FRONTS.md.
Start-up order and migrations¶
Each service that owns a database has a one-shot <name>-migrate container (restart: "no") that
runs the migrate command and exits; the service itself depends on it completing. up.sh also
converges Postgres roles on an existing host before any migration (provision-upgrade.sh),
because the provisioning script runs only on an empty data directory.
Data and secrets¶
| Where | What | Notes |
|---|---|---|
Volume postgres-data |
Every database | One cluster; one database and role per service. max_connections=80 bounds the sum of service pools |
Volume blobs |
Config's content-addressed documents and packs | Config writes; Patch mounts it read-only |
Volume recordings |
Replays and other recorded content | Patch owns it outright |
Volume valkey-data |
Session and queue state | Not durable state; losing it logs players out |
deploy/.env |
Settings and every password | Generated; never committed |
deploy/secrets/ |
Signing keys, the TOTP key, the root password, the device pepper, service keys, the player JWKS | Generated; never committed; back them up |
Passwords arrive through .env and are therefore visible to anyone who can run docker inspect
on the host. That is the accepted shape for a single host.
Images, tags and the registry¶
Every service is a 127.0.0.1:5000/otomo-<service>:<tag> image, built from its own folder (the Go services on a
distroless, non-root base; the edge and admin UI on nginx). The tag is the variable OTOMO_<SERVICE>_TAG in deploy/.env,
local by default.
| Script | Does |
|---|---|
deploy/scripts/generate-secrets.sh |
Creates .env, the Valkey config and the secrets (adds missing keys, never overwrites existing ones) |
deploy/scripts/up.sh |
docker compose up plus what YAML cannot say: provision roles, migrate, generate the auth signing key once, probe readiness |
deploy/scripts/release.sh <service> [tag] |
Gated release: pull or build the image, dump the service database when migrations are pending, run its migration before the new code, recreate the service, probe /readyz of it and its dependants, smoke test, append to deploy/releases.jsonl; rolls back automatically when that is safe (deploy/docs/RELEASE.md) |
deploy/scripts/rollback.sh --service <svc> --to <tag|release_id> |
Roll a service back by tag; refused past a migration marked -- otomo:compat breaking (deploy/docs/ROLLBACK.md) |
deploy/scripts/tunnel.sh |
Open the SSH forward to the staff plane |
deploy/scripts/backup.sh |
Write one encrypted backup archive and copy it off the host |
deploy/scripts/restore.sh |
Decrypt, verify and restore an archive into scratch or staging (deploy/docs/RESTORE-RUNBOOK.md) |
deploy/scripts/check-compose.sh |
The CI check of the rendered Compose file's invariants |
Every script that runs Compose needs a build profile (OTOMO_PROFILE=dev|staging|production, or
/etc/otomo/profile written once per host by deploy/scripts/set-profile.sh); the profile supplies
the public host and the channels, and nothing defaults (deploy/docs/PROFILES.md).
Rollback is by tag, not a rebuild. Tags are immutable and nothing prunes the registry, so
rollback.sh re-points a service at an earlier tag. Every migration declares on its first line
whether the previous release still runs on the schema it leaves (-- otomo:compat backward or
breaking); past a breaking one, a rollback needs the pre-release dump release.sh took.
The distroless images have no shell, so there are no Compose healthchecks for them. Readiness is
polled from outside against each service's :9090/readyz.
How changes are tested and shipped¶
| Where | What | When |
|---|---|---|
| GitHub Actions | Services CI (gateways, gatepolicy, admin UI, edge, and the Compose invariants), Backend services CI (gofmt, vet, -race and tests for each service against throwaway Postgres and Valkey), Matchmaker stack (Session, Allocator, Matchmaker and Gateway built and run together), Game boundary (a game's vocabulary must not reach the platform), Hygiene (rejects tool-generated footers and co-author trailers in PR text and commits) |
On pushes and pull requests that touch the matching paths |
| Jenkins | An otomo-dispatcher job receives the push webhook for staging, works out which services changed and starts the matching otomo-<service> job. Each runs tests, a smoke test, an image build and the pre-deploy levels (sanity, functional, integration, security, scaling) in a per-build Compose stack. A separate otomo-wiki-docs job rebuilds this documentation |
After a merge into staging |
Branches: pull requests go to staging; main is the release branch the quick start deploys
from. Jenkins tests and builds but does not deploy services; someone updates the host by hand
with up.sh or release.sh. Only the technical wiki redeploys itself: a push to the docs branch that touches
docs/ or services/wiki/ runs deploy/docs/deploy-docs.sh, which builds with
mkdocs --strict (a broken link or a nav page that does not exist fails the build) and replaces
the running site only if the build succeeds.
Channels are not environments
The release channels dev, staging and live are streams of content inside one stack.
They are not separate hosts. Likewise AUTH_ENV (dev, staging, production) only gates
development-only features such as the fixture login. How many hosts a team runs, and which
branch each follows, is that team's choice; this repository describes one stack.
Backups¶
deploy/scripts/backup.sh writes one archive: a pg_dump -Fc of every database (discovered
from the cluster, not listed in the script), the roles, deploy/secrets, deploy/.env and a
manifest naming the git commit, the image tags and the images running. It runs while the stack is
up and is encrypted (age or gpg to a public key whose private half stays offline). A systemd timer
(deploy/systemd/otomo-backup.timer) runs it daily; a failure alerts through the journal, a
Prometheus rule and an optional webhook.
- Retention: 7 daily and 4 weekly archives, in
~/otomo-backups/by default. - Sensitive: an archive holds the private keys and every password. It is written
0600in a0700directory and should be treated likedeploy/secrets. - Off-host copy:
OTOMO_BACKUP_OFFSITE(rclone:,rsync:,dir:or an explicitnone) in/etc/otomo-backup.conf; it has no default because a destination is a policy decision. - Tested restore:
deploy/scripts/restore.shrestores and verifies an archive, anddeploy/scripts/test-backup-restore.shruns the whole round trip against a local Postgres. - Not covered: the registry (rebuildable from the commit in the manifest), Valkey, the 72-hour observability history, and point-in-time recovery. Archives are nightly logical dumps, so the newest may be up to 24 hours old.
A lost device pepper strands every device login, and a lost signing key invalidates every token;
both are in deploy/secrets, which is why it is in the archive. Details: deploy/docs/BACKUP.md;
the step-by-step operator guide is Back up and restore otomo.
Observability¶
Metrics and logs are collected inside the stack and reach staff only through the Dashboard.
| Piece | Job | Retention |
|---|---|---|
| Prometheus | Scrapes every service's :9090/metrics every 15 s, plus node_exporter (host), cadvisor (containers) and postgres_exporter |
7 days or 2 GB |
| Loki | Log store on the local filesystem | 72 hours |
| Alloy | Tails the stack's container logs, keeps only service and level as labels, pushes to Loki |
not applicable |
| Dashboard | Runs only named query templates against Prometheus and Loki; the browser never reaches either. Serves the overview, per-service series, log search and the SSE log tail, and the merged audit | not applicable |
Metric and log names follow one convention: <service>_<thing>_<unit> metrics, JSON log lines
with a request_id. To trace a failure, copy the request_id from an error body into the admin
website's Logs page.
Alert rules live in deploy/observability/alerts.yml and show on the Overview page as firing
alerts. They include PlayerServiceDown, NoFreeGameServers, GameServerPoolEmpty,
AccountCreationSpike, RefreshTokenReuseDetected, SigningKeyRotationUnfinished, the gate alerts in
Live ops, and MatchmakerDown, MatchmakerQueueStalled and MatchmakerNoCapacity.
Prometheus and Loki publish no port. For debugging, run a query inside the container
(docker compose exec prometheus wget -qO- 'http://localhost:9090/api/v1/query?query=up') or start
a throwaway socat container on loopback and forward it over SSH (deploy/README.md has both).
Alloy mounts the Docker socket
Log discovery reads /var/run/docker.sock, which is root-equivalent on the host. It is accepted
because Alloy publishes no port and runs inside the same trust boundary as the deploy account.
Admin access¶
The staff plane is reachable only through an SSH tunnel to the host's loopback address:
One console can also operate a second environment, for example the test copy of the game on
another host: gateway_dev relays /env/<name>/... to that environment's gateway_dev over an
SSH forward (deploy/environments.example.yaml, GATEWAY_DEV_ENVIRONMENTS_FILE). The registry is
optional and absent by default; with it unset nothing changes. The relay opens no new port: the
forward is the existing tunnel's, listening on the host's loopback. Every change in a production
environment asks for the word production in the console. See
deploy/docs/ADMIN-ACCESS.md ("Operating more than one environment").
Use Chrome or Firefox: Safari refuses the Secure cookie on http://localhost. The first sign-in
uses the root password in deploy/secrets/admin_auth/root_password to create personal accounts
(deploy/scripts/rotate-root-password.sh rotates it). Step by step:
Use the admin website and deploy/docs/ADMIN-ACCESS.md.
Sizing and known constraints¶
Measured on the original 2-vCPU, 3.8 GB host, the stack idles at a few hundred MB. CPU is the
real constraint, not memory: builds and go test -race on the same host saturate both cores, so
do not release during a busy match. Docker build caches grow with every build and nothing prunes
them. Postgres is almost the whole memory footprint.
Related pages¶
- How otomo works: the big picture
- Deploy otomo on a VM: the first install.
- Servers and containers: what a container and an image are.
- Use the admin website: the SSH tunnel step by step.
- Wikis and sites: how this documentation is built and deployed.
- Troubleshooting
- Repository:
deploy/README.md,deploy/docs/BACKUP.md,deploy/docs/ADMIN-ACCESS.md,deploy/docs/EDGE-FRONTS.md,ci/services/README.md,design/00-common-stack.md.