Skip to content
Felucca Online

Docs

Server cluster: load and chaos results

How the cluster of decision #87 holds up under uo-loadtest scenario (architecture: docs/server-architecture.md). All runs on the M1 laptop (8 cores, 16 GB) while it was busy (load average 6–29 from other apps and sessions), bots and nodes on the same machine, so the absolute latencies are pessimistic and only comparable within one run (#28/#49).

How to run

cargo build --release -p uo-server -p uo-loadtest -p uo-client
# store: Apache Cassandra 5 (Homebrew) or ScyllaDB (deploy/docker-compose.yml)
MAX_HEAP_SIZE=1500M cassandra -f            # CQL on 127.0.0.1:9042
./target/release/uo-loadtest scenario --store scylla://127.0.0.1:9042 --sims 3 --gateways 2 \
    --bots 1000 --ramp 40 --duration 170 \
    --chaos "70:kill:sim-3,85:start:sim-3,105:kill:gw-1,110:start:gw-1,125:rolling" \
    --out target/loadtest/run
# without a database: --store inproc (nodes in one process on one in-memory store, real TCP)
# real game clients alongside: --clients 2 --client-cmd "UO_FRAME_CAP_MS=50 ./target/release/uo-client
#     --headless --server {urls} --name {name} --bot combat-auto --screenshot {out}/{name}.png --seconds 90"

The bot mix (--mix) defaults to combat 50, wander 30, duel 8, siege 6, churn 6 (churn bots drop their connection every 30–120 s and resume on the other gateway). Chaos: kill = SIGKILL (crash), stop = SIGTERM (graceful drain), start, rolling = every sim in turn stopped gracefully and started again. The report (report.json) has the invariant results, bot statistics, per-node metrics and a per-second series.

M7: the plan's full scenario (final)

ScyllaDB 6.2 from deploy/docker-compose.yml; 3 sim and 2 gateway processes, 2000 protocol bots and 2 real Bevy clients (--bot combat-auto, headless); chaos: SIGKILL sim-3 at 100 s and start it at 115 s, SIGKILL gw-1 at 140 s and start it at 145 s, then a rolling restart of every sim from 170 s. Report: docs/server-m7-report.json.

Target (plan §11)Result
sim tick p99 < 25 ms4.1 / 4.1 / 3.1 ms over the whole run (p50 1.5–1.8 ms, p99.9 6 ms; worst 1-s window 20 ms)
failover < 7 s4.7 s (other runs 3.6–5.2 s)
zero invariant violations0 duplicates (156 transient: one-sample overlaps of handoffs between two /state reads, mostly during the rolling drains); every region owned again
bots back after losses2000 / 2000 online at the end; 1338 disconnects, 1338 resumes, 0 failed attempts
rolling restart1.7–2.0 s per sim, no dropped connections
real clientsboth exit 0 after 240 s; kills, deaths, respawns, step prediction (49 steps, 0 rollbacks), through the sim crash and the gateway crash

Other figures of the run: 70 000 bot attacks, 7 000 region crossings, handoff p50 8–10 ms / p99 100–200 ms, store p50 6 ms / p99 60–160 ms on the single-node Scylla in Docker, command reply p50 8 ms (p99 4 s are the commands sent during a failover). Load average on the laptop 12–20 during the run.

Earlier results

Map 1024² = 256 regions. "Dups" = a player simulated by two nodes in two consecutive samples (must be 0); "transient" = seen on two nodes in one sample, a handoff caught between two /state reads.

RunStoreBotsSims / gwsDups (transient)FailoverTick p99, median / worst 1-s windowHandoffs (p50 / p99)Reconnects (failed attempts) / p99Online at end
chaos, processesCassandra 510003 / 20 (1)7.0 s8 / 262 ms2 219 (– / 33–65 ms)619 (0) / 131 ms1000
chaos, in processmem10003 / 20 (2)7.1 s16 / 33 ms5 766 (2 / 33 ms)615 (158) / 524 ms1000
chaos, in processmem20003 / 20 (6)6.1 s16 / 131 ms12 644 (4 / 65 ms)1 294 (191) / 524 ms2000
gateway killmem10003 / 20 (0)–16 / 33 ms1 117 (2 / 16 ms)513 (0) / 131 ms1000
real clients + chaosCassandra 5200 + 2 Bevy3 / 20 (0)7.0 s–––200 + 2

Chaos in the big runs: SIGKILL a sim at 70 s and start it again at 85 s, SIGKILL a gateway at 105 s and start it at 110 s, then a rolling restart of every sim (graceful drain 0.2–2 s per sim with ~300–700 players on it, no dropped connections).

Other numbers from the runs:

  • Commands: reply p50 0.5 ms, p90 4–8 ms; p99 ≈ 4 s comes from the commands sent during a failover window (they wait for the region's new owner).
  • Store (Cassandra, 3 sims): p50 1 ms, p99 33–65 ms, 0 errors; after a run it holds 1000 characters, 1000 owners and ≈ 2 200 damaged or broken objects. The contract test passes on Cassandra 5 (UO_SCYLLA=127.0.0.1:9042 cargo test -p uo-server scylla_contract).
  • Traffic: ≈ 4.5 KB/s per player to clients (2000 bots: 2.2 GB in 240 s), close to the 3.6 KB/s estimate of the proposal; node-to-node ≈ 2.2 MB/s for 2000 players.
  • Combat: 2000 bots made ≈ 78 000 attacks and ≈ 7 600 region border crossings in 4 minutes; ≈ 950 000 combat events were simulated.
  • Gateway flush p99 16–33 ms for 1000–2000 sessions on two gateways.
  • Real Bevy clients (--bot combat-auto) found camps, killed bandits, and rode through a sim crash and a gateway crash (they resumed on the other gateway); screenshots in the run's output directory.

Bugs the harness found (fixed)

From the full M7 runs on ScyllaDB:

  • A sim's tick thread panicked on a character index entry pointing at a region that had just been drained away — and the process kept renewing the leases of regions nobody simulated (146 regions dead). Removing a region purges its entries, and a dead core now gives up its membership and exits.
  • Stopping sims were still ranked, so others drained regions straight back to them: rolling restarts hit their 20 s timeout. Heartbeats now carry stopping.
  • Every sim renewed each of its ~90 leases by LWT every second inside the heartbeat loop; Paxos latency on a busy Scylla reached a second and all sims fenced all regions at once (441 expirations on one sim). Membership fencing, 10-s leases renewed every 3 s and takeover of dead nodes' regions (#90) removed it (0 expirations since).
  • A gateway restart ran CREATE KEYSPACE/TABLE IF NOT EXISTS, a cluster-wide schema operation that stalled LWT long enough to fence every sim; DDL now runs only when tables are missing.
  • Tick spikes were the thread being preempted (20–300 µs of CPU in 10–60 ms of wall time) and, rarely, path searches to unreachable or surrounded targets (up to 30 000 nodes, 16 ms of CPU). Arenas now fail such searches at once; the tick runs under a time-constraint policy.

From the earlier runs:

  • A handoff between two regions of one node lost the character's new region from the index when the accept arrived, so attaches and commands were forwarded to the node itself forever (142 922 route hops in one run) and a forced attach created a second copy (the only real duplicate ever seen).
  • Gateways located characters by their last checkpoint, which can be a handoff behind; another sim then claimed a character a live node was simulating. Fixed by routing through the live owner first.
  • Drains never finished when a handoff out of a frozen region was rejected or a player arrived late; players left behind are now re-sent.
  • Each step of every region battle scanned all of its obstacles (hundreds of trees) for timers; path finding used SipHash.
  • Region content was regenerated on the tick thread for every spawn point.
  • token is a reserved word in CQL (the sessions table).

Capacity estimate

At M7 load (≈ 700 players and 85 regions per sim) a sim's tick costs p50 1.5–1.8 ms and p99 4 ms of wall time. Earlier, the sim thread was idle ~96 % of the time at 400 players on one node (profile: rayon workers waiting), with a median tick of 2 ms for 256 regions. Ticks are parallel across regions, so a sim node scales with its cores; at the measured cost a node of 8 cores holds a few thousand players per region spread. For 30 000 players the proposal's sizing holds: ≈ 10–15 sims, 5–6 gateways (≈ 5 000 sessions each; a gateway flushed 1000 sessions with p99 16 ms) and a 3-node Cassandra/Scylla ring (the store did ~2 k writes/s here with p50 1 ms). This is an extrapolation: the laptop can't run 30 000 bots, and a single dense region is limited by one thread (one battle per region).

Environment notes

  • docker pull hangs on this machine because Docker Desktop's credential helper (docker-credential-desktop) never answers. The Scylla image was fetched on the host with crane and an empty DOCKER_CONFIG, then docker loaded: DOCKER_CONFIG=$(mktemp -d) crane pull --platform linux/arm64 scylladb/scylla:6.2 scylla.tar && docker load -i scylla.tar. After that docker compose -f deploy/docker-compose.yml up -d works as written. Cassandra 5 from Homebrew also passes the contract test.
  • macOS limits open files per process; nodes and the load tester raise their limit to the hard maximum at start (kern.maxfilesperproc 61 440).