Live ops: maintenance, update gates and announcements¶
"Live ops" is changing a running game without deploying code. This page explains the machinery behind the two controls that can stop players (the maintenance gate and the update gate), how announcements reach players, and what staff can do in the admin website today.
If you only want the clicks, see Use the admin website.
Two gates, kept apart¶
They look alike because both can refuse a request, but they answer different questions.
| Update gate | Maintenance gate | |
|---|---|---|
| Question | Is this build of the game allowed to play? | Is the service open right now? |
| Answer to a refused client | 426 client_outdated |
503 maintenance with Retry-After |
| Source of truth | The release's client_update block (versioned and rolled back with the release), plus the operator's own rules |
A record that staff edit in Ops, effective within seconds |
| Authored in | The Client update page (release) and the Update gate page (Ops) | The Maintenance page |
| Modes | off, warn, enforce (a deployment setting) |
off, warning, hard (a record), plus off or enforce as a deployment setting |
| Audit | The release audit in Config, and Ops' audit | Ops' audit |
Do not combine them: a content publish must never change maintenance state, and a maintenance toggle must never change a release.
The ops service¶
ops owns the maintenance record and the operator's update gate, caches each channel's release
policy, and serves all three to the services that enforce them as one snapshot.
staff ──admin UI──▶ gateway_dev ──▶ ops ◀── Patch ◀── Config publish
│ (long-poll per channel: "what is the live policy now?")
snapshot (ETag = revision, built every 5 s)
│
┌────────────────────────┼────────────────────────┐
▼ ▼ ▼
gateway auth session
(every player (login, refresh) (player routes)
request)
Key properties of the design:
- Enforcers pull. Gateway, Auth and Session fetch
GET /internal/ops/snapshotwith a conditional GET (every 30 s by default), each with its own service key. Ops never calls them. The shared libraryservices/gatepolicymakes the decision, so a preview in the admin website and an enforcer agree by construction. - Fail open on availability, closed on content. Before an enforcer has ever fetched a snapshot it is unarmed and lets everything through, so deploying Ops cannot cause an outage. Afterwards, a failed fetch keeps the last snapshot, so an Ops outage never lifts a gate that was active.
- One revision counter. Every change bumps a sequence, and the snapshot's
revisionis the highest. An equal revision means an identical policy. - Time is computed, not scheduled. The effective mode is a pure function of the record and the clock (to the minute), so a missed tick or a restart can never leave a stale mode. Enforcers correct for a clock offset against Ops, within limits.
Enforcement is off until you turn it on
GATE_ENFORCE_CLIENT_VERSION and GATE_ENFORCE_MAINTENANCE both default to off in
deploy/.env. With both off nothing is refused; the enforcers only count what they would
have refused (gate_decisions_total). The update gate has a warn step (count and log, let
through); stay on it for a week before enforce. One value is shared by Gateway, Auth and
Session so they cannot disagree.
Route classes and exemptions¶
Every route in every enforcing service is tagged with a class, and the gates talk about classes, never about game paths. A service refuses to start if a route has no class or one that is not in the registry, so "forgot to classify this route" fails at boot.
| Class | Examples | In a hard maintenance window |
|---|---|---|
patch.read |
Manifests, blobs | Never blocked |
auth.refresh |
Refresh, logout | Never blocked |
admin.read, admin.write, internal |
Staff routes, service-key routes | Never blocked, so the console that ends the window always works |
auth.login |
POST /auth/anonymous |
Blocked only if the record sets block login, whatever the exemptions |
session.boot |
Profile, presence heartbeat | Blocked unless exempt |
session.read, session.write |
Friends, party, events, mutations | Blocked unless exempt |
match.start |
Party launch, matchmaking queue | Blocked unless exempt |
economy.read, economy.write |
Wallet, shop, gacha | Blocked unless exempt |
telemetry.write, diagnostics.upload |
Telemetry batches and crash uploads (the optional telemetry service) | Blocked unless exempt |
match.result |
Reserved class with no routes today | Blocked unless exempt |
Staff tokens bypass the gate. The decision rules, in order, are: always-exempt classes are
allowed; staff are allowed; in an effective hard window a class is blocked unless the record
exempts it; in warning (and for an exempt class inside a hard window) the request passes with
warning headers; otherwise it is allowed. Exempting a write class needs the admin role, and
the exemptions are chosen per window, so a content deployment can keep session.read open while a
database migration keeps only patch.read.
A maintenance record also holds a notice (a title and message per language, with a stable
notice_id), a window (whole minutes, at most 24 hours), and a warning lead time. A notice id
is created with the window and kept across edits so clients can tell a new notice from the same one.
The update gate in detail¶
There are two inputs, and an enforcer applies the stricter one for a platform:
- The release's
client_updateblock, authored per release on the Client update page. It holds the latest version, the minimum per platform, the action to take (open a link), a message per language and support and store links. It travels with the release, so it versions and rolls back with it. Config validates and previews it before publishing, and republishing only the gate block is alive_opsaction on any channel (changing the minimums and latest version staysadmin). - The operator's gates in Ops, authored on the Update gate page: for a channel and a platform, the oldest build, where to send the player, what to tell them, and whether to refuse or only warn.
Patch exposes the live policy on /patch/v1/live/client for launchers, and Ops keeps a cached
copy per channel by long-polling Patch (a poll every 30 s corrects anything the watch misses;
ops_update_cache_drift_total counts misses). Clients identify their build with the
X-Otomo-Client-Version header. Auth runs the check at login and refresh, so a too-old build
cannot even sign in.
Announcements¶
Announcements are not a service. They are a list in a client-audience Config namespace
named announcements: one row per announcement per language, with a published flag, an optional
start and end, and an order. The admin website's Announcements page edits that namespace's
draft only; the changes reach players the normal way, by creating a version and publishing a
release. As a result an announcement is audited by Config, goes through dev, staging and
live, and rolls back with the release it shipped in. The page previews what a player would see
at a time you choose.
The game reads the namespace through the SDK's RemoteConfig. See
Namespace presets for the schema.
What staff can do in the admin website today¶
The menu is derived from the same table that guards the routes, so a role sees only its pages.
| Section | Page | Minimum role to view | Changes need |
|---|---|---|---|
| Live ops | Namespaces, Schema, Packs | viewer |
Drafts and versions, packs: live_ops. New namespaces, schemas: admin |
| Live ops | Releases, Compose release | viewer |
Publish to dev and staging: live_ops. To live, roll back, promote: admin. The live target asks you to type live |
| Live ops | Maintenance | viewer |
live_ops. Exempting write classes: admin |
| Live ops | Update gate | viewer |
live_ops |
| Live ops | Client update | viewer |
Message and links: live_ops. Minimums: admin |
| Content | Announcements | viewer |
live_ops (draft); a release publishes it |
| Content | Curated recordings | viewer |
The patch.curation draft, then a release |
| Players | Players, one player's page and timeline | viewer |
Disband party: live_ops, with a reason of 8+ characters. The force-disband is the only staff action Session has |
| Players | Player economy: wallet, ledger, entitlements | viewer |
Grant reward by reward class: live_ops, reason required. There is no revoke |
| Economy | Reconciliation | viewer |
Read only |
| Services | Overview, Logs, service detail | viewer |
Read only |
| Audit | Audit (merged feed from Config, admin-auth and Session) | viewer |
Read only. Ops keeps its own audit (GET /api/admin/ops/audit), and Economy records staff grants in the ledger |
| Admin | Users | admin |
Invite, change roles, send resets (not for other admins: root only) |
| any | Account | any signed-in staff | Password, sessions, two-factor |
Every write is attributed to the staff account in an audit row. Destructive actions on live
ask for confirmation by typing live, and the UI shows the audit entry afterwards.
Planned
Config proposals with approval, namespace archiving and further release safeguards are planned and are not part of what this page describes.
Observing the gates¶
Prometheus alerts in deploy/observability/alerts.yml cover the gate machinery: OpsServiceDown,
GateSnapshotNeverFetched, GateSnapshotStale, OpsSnapshotBuildStalled, UpdateCacheDrift,
ClockOffsetHigh and MaintenanceActive. See
Deployment and operations.
Related pages¶
- How otomo works: the big picture
- Use the admin website: sign-in and a tour.
- Config, Patch and content delivery: releases and channels.
- Identity and sign-in: the role ladder.
- Namespace presets: the announcements schema.
- Error codes:
426and503 maintenance. - Repository:
design/23-update-gate.md,design/25-ops-gates.md,services/ops/README.md,services/ops/openapi/admin-ops.yaml,services/gatepolicy/README.md.