Skip to content

Live ops: maintenance, update gates and announcements

"Live ops" is changing a running game without deploying code. This page explains the machinery behind the two controls that can stop players (the maintenance gate and the update gate), how announcements reach players, and what staff can do in the admin website today.

If you only want the clicks, see Use the admin website.

Two gates, kept apart

They look alike because both can refuse a request, but they answer different questions.

Update gate Maintenance gate
Question Is this build of the game allowed to play? Is the service open right now?
Answer to a refused client 426 client_outdated 503 maintenance with Retry-After
Source of truth The release's client_update block (versioned and rolled back with the release), plus the operator's own rules A record that staff edit in Ops, effective within seconds
Authored in The Client update page (release) and the Update gate page (Ops) The Maintenance page
Modes off, warn, enforce (a deployment setting) off, warning, hard (a record), plus off or enforce as a deployment setting
Audit The release audit in Config, and Ops' audit Ops' audit

Do not combine them: a content publish must never change maintenance state, and a maintenance toggle must never change a release.

The ops service

ops owns the maintenance record and the operator's update gate, caches each channel's release policy, and serves all three to the services that enforce them as one snapshot.

 staff ──admin UI──▶ gateway_dev ──▶ ops ◀── Patch ◀── Config publish
                                       │       (long-poll per channel: "what is the live policy now?")
                       snapshot (ETag = revision, built every 5 s)
                                       │
              ┌────────────────────────┼────────────────────────┐
              ▼                        ▼                        ▼
           gateway                    auth                    session
        (every player              (login, refresh)        (player routes)
         request)

Key properties of the design:

  • Enforcers pull. Gateway, Auth and Session fetch GET /internal/ops/snapshot with a conditional GET (every 30 s by default), each with its own service key. Ops never calls them. The shared library services/gatepolicy makes the decision, so a preview in the admin website and an enforcer agree by construction.
  • Fail open on availability, closed on content. Before an enforcer has ever fetched a snapshot it is unarmed and lets everything through, so deploying Ops cannot cause an outage. Afterwards, a failed fetch keeps the last snapshot, so an Ops outage never lifts a gate that was active.
  • One revision counter. Every change bumps a sequence, and the snapshot's revision is the highest. An equal revision means an identical policy.
  • Time is computed, not scheduled. The effective mode is a pure function of the record and the clock (to the minute), so a missed tick or a restart can never leave a stale mode. Enforcers correct for a clock offset against Ops, within limits.

Enforcement is off until you turn it on

GATE_ENFORCE_CLIENT_VERSION and GATE_ENFORCE_MAINTENANCE both default to off in deploy/.env. With both off nothing is refused; the enforcers only count what they would have refused (gate_decisions_total). The update gate has a warn step (count and log, let through); stay on it for a week before enforce. One value is shared by Gateway, Auth and Session so they cannot disagree.

Route classes and exemptions

Every route in every enforcing service is tagged with a class, and the gates talk about classes, never about game paths. A service refuses to start if a route has no class or one that is not in the registry, so "forgot to classify this route" fails at boot.

Class Examples In a hard maintenance window
patch.read Manifests, blobs Never blocked
auth.refresh Refresh, logout Never blocked
admin.read, admin.write, internal Staff routes, service-key routes Never blocked, so the console that ends the window always works
auth.login POST /auth/anonymous Blocked only if the record sets block login, whatever the exemptions
session.boot Profile, presence heartbeat Blocked unless exempt
session.read, session.write Friends, party, events, mutations Blocked unless exempt
match.start Party launch, matchmaking queue Blocked unless exempt
economy.read, economy.write Wallet, shop, gacha Blocked unless exempt
telemetry.write, diagnostics.upload Telemetry batches and crash uploads (the optional telemetry service) Blocked unless exempt
match.result Reserved class with no routes today Blocked unless exempt

Staff tokens bypass the gate. The decision rules, in order, are: always-exempt classes are allowed; staff are allowed; in an effective hard window a class is blocked unless the record exempts it; in warning (and for an exempt class inside a hard window) the request passes with warning headers; otherwise it is allowed. Exempting a write class needs the admin role, and the exemptions are chosen per window, so a content deployment can keep session.read open while a database migration keeps only patch.read.

A maintenance record also holds a notice (a title and message per language, with a stable notice_id), a window (whole minutes, at most 24 hours), and a warning lead time. A notice id is created with the window and kept across edits so clients can tell a new notice from the same one.

The update gate in detail

There are two inputs, and an enforcer applies the stricter one for a platform:

  1. The release's client_update block, authored per release on the Client update page. It holds the latest version, the minimum per platform, the action to take (open a link), a message per language and support and store links. It travels with the release, so it versions and rolls back with it. Config validates and previews it before publishing, and republishing only the gate block is a live_ops action on any channel (changing the minimums and latest version stays admin).
  2. The operator's gates in Ops, authored on the Update gate page: for a channel and a platform, the oldest build, where to send the player, what to tell them, and whether to refuse or only warn.

Patch exposes the live policy on /patch/v1/live/client for launchers, and Ops keeps a cached copy per channel by long-polling Patch (a poll every 30 s corrects anything the watch misses; ops_update_cache_drift_total counts misses). Clients identify their build with the X-Otomo-Client-Version header. Auth runs the check at login and refresh, so a too-old build cannot even sign in.

Announcements

Announcements are not a service. They are a list in a client-audience Config namespace named announcements: one row per announcement per language, with a published flag, an optional start and end, and an order. The admin website's Announcements page edits that namespace's draft only; the changes reach players the normal way, by creating a version and publishing a release. As a result an announcement is audited by Config, goes through dev, staging and live, and rolls back with the release it shipped in. The page previews what a player would see at a time you choose.

The game reads the namespace through the SDK's RemoteConfig. See Namespace presets for the schema.

What staff can do in the admin website today

The menu is derived from the same table that guards the routes, so a role sees only its pages.

Section Page Minimum role to view Changes need
Live ops Namespaces, Schema, Packs viewer Drafts and versions, packs: live_ops. New namespaces, schemas: admin
Live ops Releases, Compose release viewer Publish to dev and staging: live_ops. To live, roll back, promote: admin. The live target asks you to type live
Live ops Maintenance viewer live_ops. Exempting write classes: admin
Live ops Update gate viewer live_ops
Live ops Client update viewer Message and links: live_ops. Minimums: admin
Content Announcements viewer live_ops (draft); a release publishes it
Content Curated recordings viewer The patch.curation draft, then a release
Players Players, one player's page and timeline viewer Disband party: live_ops, with a reason of 8+ characters. The force-disband is the only staff action Session has
Players Player economy: wallet, ledger, entitlements viewer Grant reward by reward class: live_ops, reason required. There is no revoke
Economy Reconciliation viewer Read only
Services Overview, Logs, service detail viewer Read only
Audit Audit (merged feed from Config, admin-auth and Session) viewer Read only. Ops keeps its own audit (GET /api/admin/ops/audit), and Economy records staff grants in the ledger
Admin Users admin Invite, change roles, send resets (not for other admins: root only)
any Account any signed-in staff Password, sessions, two-factor

Every write is attributed to the staff account in an audit row. Destructive actions on live ask for confirmation by typing live, and the UI shows the audit entry afterwards.

Planned

Config proposals with approval, namespace archiving and further release safeguards are planned and are not part of what this page describes.

Observing the gates

Prometheus alerts in deploy/observability/alerts.yml cover the gate machinery: OpsServiceDown, GateSnapshotNeverFetched, GateSnapshotStale, OpsSnapshotBuildStalled, UpdateCacheDrift, ClockOffsetHigh and MaintenanceActive. See Deployment and operations.