Skip to content

Relay capacity: first measurements (design 31 P1)

What was measured: the Gameplay Proxy in relay mode (PROXY_RELAY_ENABLED=true) carrying peer-hosted rooms, driven by the Go bots in services/gameplay_proxy/cmd/relaybot (internal/relaybot). Every number below is from short runs on a developer PC, not from staging or the NAS. Treat them as an upper bound on cost per datagram and a sanity check of design 31 §3.6, not as the P1 exit soak (2 h at 25 rooms on staging), which is still to do.

Machine and method

CPU AMD Ryzen 5 3600X, 6 cores / 12 threads, 3.8 GHz
RAM 15.9 GB
OS Windows 10 Home; Go toolchain from ~/tools/go-dist
Network loopback only (127.0.0.1); no NIC, no NAT, no real internet RTT
Bots same machine as the proxy: one host bot + (players − 1) client bots per room, 30 Hz each way, 200-byte payloads
Proxy separate process (gameplay_proxy.exe), so its CPU and memory are read apart from the bots' (Get-Process before and after the measured window)
Allocator relaybot's fake Allocator (real Ed25519 claims, JWKS, directory with rooms, heartbeat/ended/events routes)
Latency one-way, bot send → bot receive, from the Windows performance counter (Go's own monotonic clock on this machine steps in 1 ms)

Reproduce (two shells, both bound to 127.0.0.1):

relaybot -proxy 127.0.0.1:27100 -allocator 127.0.0.1:18080 -key-file key.txt -rooms 25 -players 4 -duration 60s
PROXY_LISTEN_ADDR=127.0.0.1:27100 PROXY_METRICS_ADDR=127.0.0.1:19090 \
PROXY_ALLOCATOR_URL=http://127.0.0.1:18080 PROXY_ALLOCATOR_KEY_PATH=key.txt \
PROXY_RELAY_ENABLED=true PROXY_DIRECTORY_REFRESH=1s gameplay_proxy

relaybot with no -proxy runs the relay in-process instead (bots and proxy share one process, so CPU is not separable). It refuses an offered load above -max-mbps (default 5).

Results

A. Design target: 25 rooms × 4 players, 60 s measured

to host to clients
datagrams sent / received 133,984 / 133,966 133,977 / 133,970
loss 0.013 % 0.005 %
one-way latency p50 / p99 131 µs / 12.0 ms 198 µs / 9.1 ms
  • Relayed 4,463 datagrams/s in total (≈ 3.6 Mbit/s of payload each way).
  • Proxy CPU: 1.14 s over 62.4 s ≈ 1.8 % of one core (≈ 4 µs per relayed datagram).
  • Proxy memory: Windows working set 61 MB, Go heap in use 3.7 MB. An idle relay-mode proxy started the same way measured 10.8 MB working set in run B.
  • Cross-room or cross-slot deliveries: 0. Relay drops (any reason): 0.
  • The "loss" is at the edges of the measured window: a datagram sent just before the window closed and received just after is counted as sent but not received, or the other way round. Run B shows the same thing as −0.007 %.

B. Stress: 50 rooms × 8 players, 20 s measured (PROXY_MAX_ROOMS=60, -max-mbps 30)

  • Relayed 20,743 datagrams/s (≈ 16.6 Mbit/s payload each way), loss ≈ 0, leaks 0, drops 0.
  • Proxy CPU: 5.38 s over 20.4 s ≈ 26 % of one core (≈ 13 µs per datagram).
  • Working set 18 MB (peak 18.9 MB).
  • One-way p50 0.5 / 0.76 ms, p99 ≈ 98 ms.

C. Impairment profiles (in-process, 4 rooms × 4 players, 10 s each)

Profile (applied to every bot's sends) loss to host / to clients one-way p50 p99
2 % loss, 40 ms 1.6 % / 1.9 % 40.7 ms 97–105 ms
10 % loss, 150 ± 50 ms 10.0 % / 8.4 % 152–153 ms 206–211 ms

The relay adds nothing measurable to the shim's own loss and delay; the relay neither reorders nor retransmits, so ENet's own reliability is all a game gets, as designed.

Reading the numbers honestly

  • CPU is not the limit at the design target. 25 × 4 cost under 2 % of one desktop core. Run B's per-datagram cost was three times run A's; the likely causes are Windows loopback, scheduler contention from 800 bot goroutines on the same 12 threads, and cache effects, not the relay's logic, but this was not proven. A small x86 NAS core is slower than a Ryzen 3600X: budget 10–20 µs per datagram there until staging measures it.
  • The p99 is the bots', not the relay's. With every bot, its timers and the proxy on one Windows machine, the tail is scheduling: run A's p99 (9–12 ms) and run B's (≈ 98 ms) grew with the number of bot goroutines, while the p50 stayed sub-millisecond. Design 31's P1 exit criterion ("p99 relay-added latency < 2 ms") needs the bots on another machine than the proxy; it is not demonstrated by these runs.
  • Memory: the Go heap stayed under 4 MB at 25 rooms. The Windows working-set figure varied between runs (61 MB in A, 18 MB in B at a higher load) with the runtime's page reuse; Linux RSS on staging is the number that matters for design 31's "< 40 MB" exit criterion and has not been taken yet.
  • Bandwidth is the real limit, and this PC cannot show it. Loopback has no uplink. Design 31 §3.6's arithmetic stands: 25 rooms of 4 at 30 pps × 200 B is ≈ 8 Mbit/s each way on the wire, which a residential NAS uplink may not have spare. Keep PROXY_MAX_ROOMS at 10 on production until the uplink is measured (design 31 §8 P4).
  • Not measured here: a real NIC, real NAT rebinding, packet loss bursts, a 2-hour run, the Allocator's database under 25 rooms' heartbeats, and Linux.