Skip to content
Felucca Online

Docs

Game server cluster

Status: implemented (decisions #87 server, #88 online client, #90 ownership timings and tick scheduling). The on-chain ledger (chain/, #7–#9) is off: inventories live in the server's store, and Gather/Craft answer "ledger off". Load and chaos results: docs/server-performance.md.

Goal: uo-server and uo-client working against each other for real — login, walking, real-time combat (#68) and destruction (#66) — on a server built for tens of thousands of concurrent sessions across several instances sharing one Cassandra-class database, with players switching instances (region handoffs, failover, rebalancing, rolling restarts) without noticing.

cargo run -p uo-server                                   # one process, in-memory store, ws://127.0.0.1:7777
cargo run --release -p uo-client -- --server ws://127.0.0.1:7777 --name Me
cargo test -p uo-server --test cluster                   # multi-node tests (one process, real TCP)
cargo run --release -p uo-loadtest -- scenario --store inproc --sims 3 --gateways 2 --bots 1000 \
    --chaos "70:kill:sim-3,85:start:sim-3,105:kill:gw-1,110:start:gw-1,125:rolling"

1. Why the database is not the shared state

A tick is 50 ms (uo_combat::SIM_STEP). Cassandra/Scylla resolve conflicts with last-write-wins and have no multi-row transactions, so two instances moving one character would silently overwrite each other, and a read-modify-write per entity per tick costs more than the tick. So:

  • Live state is in memory with exactly one owner. The world is cut into regions; each is simulated by one sim node holding its lease.
  • The store holds durable state (characters, world object damage), ownership (leases, character owners), membership and sessions. Compare-and-set (LWT) is used only where it decides who may simulate what; state writes are cheap plain writes fenced by generation.

2. Topology

        clients / SDK bots / MCP   (public JSON protocol v2 over WebSocket)
              │            │
         ┌────▼───┐   ┌────▼───┐
gateway  │  gw-1  │   │  gw-2  │   sessions, area of interest, routing, resume
         └─┬──┬───┘   └─┬──┬───┘   (no game state)
           │  internal protocol: MessagePack frames over TCP, one link per peer
     ┌─────▼──┐ ┌───────▼┐ ┌────────┐
sim  │ sim-1  │◀▶│ sim-2  │◀▶│ sim-3  │   regions on leases, 20 Hz, uo-combat battles
     └────┬───┘ └───┬────┘ └───┬────┘
          └─────────┼──────────┘
              ┌─────▼──────┐
store         │ Scylla /   │   leases + owners (LWT), characters + objects
              │ Cassandra  │   (fenced writes), nodes, sessions (TTL)
              └────────────┘
  • One binary, --role gateway|sim|all. all + --store mem is the one-process dev server; --store scylla://host:9042 (also Cassandra) for real clusters. Nodes find each other through the nodes table (heartbeat every second, TTL 5 s) and talk directly over TCP — no broker.
  • Admin HTTP per node (--admin): /metrics (Prometheus text with p50/p99/max of tick, handoff, store and gateway-flush latencies), /state (regions and players with generations, for invariant checks), /drain?region=R&to=NODE, /shutdown (graceful drain).

3. World and regions (uo-world)

  • Map 1024×1024 by default (--map), generated per tile from the seed (uo_proto::map::tile_at, windows match the whole map exactly), cut into regions of 64×64 (256 regions). Area of interest radius 18 (UO's update range).
  • Every region has deterministic content (region_content, cached per process by cached_content): up to two bandit camps from the combat demo (camp.rs, roster uo_combat::demo::roster(Scene::Camp)) and nature by terrain, with stable object keys region << 24 | index (runtime objects set bit 23). Only the diff travels and is stored.
  • build_arena (shared with the client demo) turns content plus object state into a uo_combat::Arena: walls/fences/doors → edge obstacles, other destructible statics → tile obstacles, cover from baked sprite heights (embedded at build time).
  • Spawn points spread new characters over all regions by name hash, at least 14 tiles from camps.

4. The sim

  • A dedicated thread ticks every owned region every 50 ms and handles messages between ticks. Store calls never block it: they run as tokio tasks and post their results back. Regions tick on that thread while the summed work is light (< 8 ms), else in parallel on a pool of half the cores (UO_TICK_THREADS). On macOS the tick thread runs under the time-constraint policy (period 50 ms, 10 ms of CPU, constraint 25 ms) and the pool at user-interactive QoS, so a loaded machine doesn't preempt it (#90).
  • Path searches fail at once when no free goal tile is reachable (arena connected components, rebuilt when obstacles change), and a failed search isn't repeated from the same tile for a second: a target walled off or surrounded by a crowd used to cost a whole-arena search.
  • One open-world battle per region (Battle::open): the arena is the region plus ARENA_MARGIN = 12 tiles of its neighbours; it never ends, nobody escapes, mobs are leashed to their spawn (LEASH 16) and walk home, everyone regenerates. Moving is the same tempo system: a player runs out of a fight (a step is 1 AP = 0.25 s) and walks in one (2 AP = 0.5 s; in a fight = attacked or attacking in the last 10 s), #92. Players have sides of their own (PvP is open), mobs side 1.
  • Players are blessed warriors, archers or mages by name hash (the demo hero's +120 hits, five potions); they respawn near the region's centre 8 s after dying. Bandits' bodies stay 10 s, they return 30 s after dying.
  • Checkpoints: a changed character every 15 s and at logout; object changes every tick they happen. A disconnected character lingers 30 s.

5. The gateway

  • Login (dev mode: names are accounts, char_id = fnv(name)), a resume token stored with a TTL, then locating the character: its live owner if any (char_owner), else the lease holder of the region of its last checkpoint (or of its spawn point).
  • Subscribes to the feeds of the regions within the area of interest + 4 tiles, merges them (an entity is taken from the region that last upserted it) and sends each session a Snapshot, then per-tick Deltas and events within radius 18. JSON is encoded once per entity version and per event; per-session messages are assembled from these fragments. Sessions with nothing new around them are skipped.
  • Routes commands with a per-session seq to the owner; buffers them while the character is being located or attached.

5a. Protocol v2 (uo_proto::msg)

Login / Resume{token} → Welcome{token, map seed and size, tick_hz 20, region_size, aoi_radius, x, y, spells}; Snapshot{entities, objects} then Delta{enter, update, leave, objects}; Event::Combat{at_ms, CombatEvent} (= uo_combat::Event with entity ids, object keys, tiles); Respawned, StepRejected{x, y} (a failed step or walk: clients that predicted their own steps roll back). Orders: WalkTo (continues across regions, in legs planned by uo_world::walk, the planner the client predicts with), Move { dir } (a held heading, UO's right mouse button / WASD: one step per step time, sliding along obstacles, until dir: null, #91; the client uses it only in a fight), Step (queued, up to 4, taken in order and checked; out of a fight the client drives held running by sending each predicted step, #94), Stop, Attack, Cast, Guard, Potion, Hold, Open, Plant; Say, Look, Inventory, Equip, Unequip, Ping.

6. Switching instances

6.1 Ownership and fencing (#90)

  • Membership is the fast liveness signal: every node writes its nodes row every 500 ms with a TTL of 3 s (a plain write). A sim fences itself: it stops simulating every region if no heartbeat succeeded for 2.5 s, i.e. before its row can expire.
  • region_lease(region) → (node, epoch, expires_ms): LWT, TTL 10 s, renewed every 3 s (one Paxos round per region every 3 s; a slow store doesn't make sims drop regions). A new holder gets epoch + 1 (the epoch lives in a row that never expires). A sim also stops a region 2 s before its lease would expire unless renewed.
  • A region whose holder has dropped out of the membership is taken over at once by compare-and-set on (node, epoch): safe, because that holder fenced itself before its membership expired. Failover ≈ the membership TTL plus the region build.
  • char_owner(char) → (node, gen): LWT compare-and-set on gen. This is the commit point of every handoff and attach.
  • Character and object state are plain writes USING TIMESTAMP gen·2⁴⁰ + n (objects: the lease epoch): a former owner's late writes always lose.
  • An owner checks every 15 s (with its checkpoint) that it still owns each character at its generation, and drops a copy that someone else claimed.
  • Heartbeat, lease renewal and placement are three independent loops, so a slow store call in one never delays the others. Schema DDL runs only when tables are missing (a DDL stalls LWT cluster-wide on Scylla).
  • A sim whose tick thread panics stops heartbeating and renewing, and the process exits: its regions fail over instead of staying owned and dead.

6.2 Border handoff

A player at least HANDOFF_DEPTH = 2 tiles past its region's border, and not fighting for 10 s (or 10 tiles in, whatever), leaves its battle and waits as outgoing; the destination node claims gen + 1, checkpoints, admits it, accepts, and re-routes its gateway. Commands that arrive in between are buffered and forwarded; the destination drops any seq it has already applied. No answer in 2 s: the source reads the owner — a higher generation means it moved, else the player goes back into the source battle. Handoffs between two regions of one node take the same path.

6.3 Battle pinning

A player who attacks or casts at an entity of a neighbouring region moves into that region's battle (with the order replayed there) when it stands inside that region's arena. While fighting, geographic handoff waits. Damage to objects of a neighbour inside the margin is sent to their owner (ObjectHits); object changes are sent to the neighbours' arenas.

6.4 Failover

A crashed sim drops out of the membership after ≤ 3 s; its regions are taken over by rendezvous ranking (§6.6, rank k waits k × 1 s). Gateways see the link drop and re-attach their sessions: the new owner claims the character at its last checkpoint (≤ 15 s old). Measured on ScyllaDB under full load: 3.6–5.2 s until every region is owned again.

6.5 Planned migration, graceful stop, rolling restart

Drain(region → node): freeze, checkpoint every player and the objects, transfer the lease (epoch + 1), the new owner loads the region and reports ready, then every player is handed over (players left behind by a rejected handoff are sent again each tick; a target that never reports ready is given up after 6 s). ≈ 50 ms per region. SIGTERM (or /shutdown) drains every region to the next-ranked sim and exits; a stopping sim announces it in its heartbeat and takes no regions. Measured on ScyllaDB: 1.5–2 s per sim with ~700 players on it, no connection drops.

6.6 Placement and rebalancing

Rendezvous hashing per region over the live, non-stopping sims: the top-ranked sim takes a free region at once, rank k after k·1 s. A region whose best sim is another node that has been up for 5 s is drained to it (≤ 4 at a time), so a joining node takes its share and a restarted node gets its regions back. (The proposal ranked 2×2 blocks for locality; on small maps blocks left nodes empty, so it ranks regions.)

Demand-driven worlds (#100): a world with more than 4096 regions (the baked Felucca has 65 536) cannot keep every region leased and ticking. There a region runs only while someone needs it. Sims publish the regions their players stand in plus their 8 neighbours (so a handoff target is always running) into the store's region_demand table (TTL 20 s, refreshed every 3 s); a gateway locating a character whose region has no owner writes that region and its neighbours. The placement loop then takes only demanded regions (same rendezvous ranking), and a held region nobody has demanded for 30 s is retired: its objects are checkpointed, the lease released, and it stops (never while players are in it). Smaller worlds keep every region running, as before.

6.7 Gateway loss

Clients reconnect along their gateway list with Resume{token}; the character keeps playing meanwhile. Measured: 500+ simultaneous resumes, p99 0.5 s.

6.8 Invariants (checked every second by uo-loadtest scenario)

  1. No player is simulated by two nodes (seen on two nodes in two consecutive samples; a single sample can catch a handoff between two /state reads and is reported as transient).
  2. Every region is owned again within the failover window.
  3. Every bot that lost its node or gateway gets back in.

7. Store (uo-server/src/store)

Store trait with MemStore (exact CAS/LWW/TTL semantics; one process, or several nodes in one process for tests) and ScyllaStore (scylla driver; ScyllaDB or Apache Cassandra 4/5). Contract tests run on both (UO_SCYLLA=host:9042 cargo test -p uo-server scylla_contract).

region_lease (region int PRIMARY KEY, node text, epoch bigint, expires_ms bigint)  -- LWT
char_owner   (char_id bigint PRIMARY KEY, node text, gen bigint)                    -- LWT
characters   (char_id bigint PRIMARY KEY, row blob)                                 -- USING TIMESTAMP
objects      (region int, key bigint, row blob, PRIMARY KEY (region, key))          -- USING TIMESTAMP
nodes        (node text PRIMARY KEY, info blob)                                     -- USING TTL
sessions     (tok text PRIMARY KEY, char_id bigint, name text)                    -- USING TTL

Keyspace replication factor UO_SCYLLA_RF (default 1); on Scylla the keyspace is created without tablets (LWT). deploy/docker-compose.yml runs ScyllaDB; Homebrew's cassandra works as well.

8. Testing

  • cargo test -p uo-server --test cluster: one node (login, walk, fight), three sims + two gateways (handoffs across nodes, a sim crash, every player back), a planned drain, a gateway loss (resume < 5 s), a graceful shutdown (players handed over, no drops).
  • uo-loadtest swarm: N protocol bots (uo_sdk::bot: combat, wander, duel, siege, churn) against running gateways.
  • uo-loadtest scenario: starts the cluster (--store scylla://… as processes, --store inproc as in-process nodes on one store with real TCP between them), the swarm, optional real headless game clients (--clients 2 --client-cmd "uo-client --headless --server {urls} --name {name} --bot combat-auto --screenshot {out}/{name}.png --seconds 70"), chaos steps secs:kill|stop|start:node and secs:rolling, the invariant checks, and report.json.

9. Not done yet / next

  • Economy on chain (#7–#9): the store is the source of truth for now.
  • Mobs never cross regions (leashed); one battle per region caps a siege at one sim thread per region.
  • A binary encoding of deltas for clients (JSON is fine at the measured load). Client step prediction is done (#88): own steps start at once along the server's path and roll back on StepRejected.
  • Accounts and authentication (names are accounts in dev mode).