[FrostGod]

Explainer for reviewers: open the one-page visual (gist source). It covers current vs. proposed behavior, the cap, the stay flag, rollouts, and the one trade-off.

Screenshot

Summary

Shared supervisors now spread evenly across gateway replicas. A supervisor that comes in through the Service goes to the first gateway clockwise on the existing gateway ring that serves fewer than ⌈supervisors ÷ gateways⌉, and a stopping gateway hands its supervisors off the same way. Supervisors move only when they reconnect through the Service; there is no rebalancer.

Related Issue

Closes #5434

Changes

• Protocol: HostGatewayMessage.redirect (HostRedirect { peer_endpoint }) and HostHello.stay.
• Gateway:
  • On attach, a hello without stay is redirected, before any session starts, when the ring and cap place the member elsewhere.
  • On shutdown, each stream ends with a redirect computed on the ring without this replica.
  • Counts come from the pool member rows the reconciler already reads (MemberView.gateway_replica).
  • New helper: GatewayRing::first_below.
• Supervisor:
  • Follows a redirect at once, without backoff, with stay.
  • Also sets stay on the sticky redial (token refresh, blips) and on the Service fallback after a failed redirect, so a stale ring cannot loop it.
  • A gateway always accepts stay.
• Metrics: openshell_server_supervisor_hosts (supervisors per gateway) and openshell_server_supervisor_host_instances_max (busiest supervisor's sandboxes). The second is a max rather than a per-supervisor label because member IDs are pod names, which would make the label set grow without bound.
• Docs: HA page, metrics catalog, and the supervisor-control README.

Known trade-off: nothing moves after placement, so two cases can stay uneven until supervisors restart:

• a gateway added without a rollout;
• a supervisor rolling update with surge, where a new supervisor can't know which old one leaves next.

Gateway rollouts and crashes, and supervisor restarts, end even.

Testing

• cargo clippy -D warnings and cargo fmt --check for openshell-server, openshell-supervisor, openshell-supervisor-control, and the e2e crate
• Unit tests: 2,646 in openshell-server, 293 in openshell-supervisor, 12 in openshell-supervisor-control, all pass. New tests:
  • first_below skips full replicas and falls back to the owner
  • arrivals under the cap fill every replica evenly
  • spread_home ignores the member's own record and leaving members
  • the supervisor follows a redirect at once with stay, then falls back to the Service with stay
• E2E on a local k3d cluster (2 gateways, 2 supervisors, Postgres) with this branch's images:
  • new supervisors_spread_evenly_across_gateways (restart each supervisor, then roll the gateways, and assert one supervisor per gateway): passed 3 of 3 runs
  • pool_sessions_follow_their_supervisors_gateway: passed
  • losing_a_supervisor_pod_moves_its_sandboxes_without_restarting_them stops at its first host-endpoint call with policy_dns_trusted_gateway_unavailable, before any supervisor or gateway moves. The hand-built cluster lacks the host-gateway DNS setup the CI lane provides.

Checklist

• Follows Conventional Commits
• Commits are signed off (DCO)
• Architecture docs updated (if applicable)

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[github-actions]

🌿 Preview your docs: https://nvidia-preview-pr-5436.docs.buildwithfern.com/openshell

[FrostGod]

/ok to test 0f34dee

[github-actions]

Compatibility review (advisory)

New or updated breaking changes; see the full report for all findings.

1. rust-sdk — No explicit risk acceptance found

Under openshell_sdk::raw::proto::supervisor_host::v1, HostHello gains stay and host_gateway_message::Payload gains Redirect, invalidating formerly complete literals and exhaustive matches.

• Impact: Raw SDK consumers constructing HostHello or exhaustively matching Payload fail to compile, although ordinary RPC calls and generated server-trait signatures remain unchanged.
• Action: Preserve the baseline exported Rust shapes for v0.1.8 by isolating the extended private protocol in separate types, or defer the break to a permitted minor train with guidance for stay/default initialization and Redirect handling.
• PR: feat(supervisor): spread shared supervisors evenly across gateway replicas #5436

Review details

• Execution: complete
• Candidate: PR #5436 (44164f0c877a9626ae9c59be58688ec1fdf2efb3)
• Baseline: 4f79018659447c6589dc6a340fda44712ae8ef16 (4f79018659447c6589dc6a340fda44712ae8ef16)
• Reviewed: rust-sdk, python-sdk, go-sdk, typescript-sdk, api-behavior, cli, configuration, policy, helm, state

Advisory review; publication gates are unchanged.

Branch check run

[TaylorMutch]

/ok to test fb1e825

[TaylorMutch]

I found two correctness issues in the placement and redirect handling; details are attached inline.

Validation: ran 12 existing ring tests and two isolated reproductions using the PR's placement functions. All passed, confirming that concurrent arrivals and handoffs can exceed the intended cap. I did not run the full build or Kubernetes E2E.

[TaylorMutch]

Concurrent arrivals or handoffs can permanently defeat the placement cap. Every decision reads the cached pool view, without reserving capacity or revalidating it at the destination; redirected hosts then disable further redirects. For example, two arrivals can both see an empty view and hash to the same gateway, leaving a 2:0 distribution. With three gateways initially serving 2:1:1 supervisors, the two supervisors on a stopping gateway can also both choose the same remaining gateway from the shared snapshot, leaving 3:1 instead of 2:2. I reproduced both cases using this PR's placement functions. Previously placement depended on the Service; the new cap still leaves operators with persistent skew because accepted hosts never rebalance. Please account for in-flight placement decisions and validate capacity before making a connection sticky, and cover simultaneous arrivals/handoffs. The new sequential restart E2E does not exercise this case.

[FrostGod]

Good catch: the records lag about 2 seconds, so a burst of connects piled onto one replica. Each gateway now counts the placements it just made, and placements run one at a time. Hosts follow at most 2 redirects in a row, so the receiving gateway can correct a placement once.

[TaylorMutch]

Rechecked at 9e93a2c. The local placement accounting fixes bursts through one gateway, but I reproduced a remaining race across gateways before the next observation. A host accepted after two redirects sets accepts_redirect = false, so the controller skips home() and the actual destination never records that acceptance in its local SupervisorSpread.

With three gateways and three hosts, this schedule settles at 2:1:0 instead of 1:1:1:

1. sup-2 arrives at gw-2, which redirects it to gw-1.
2. sup-1 arrives at gw-1 and is accepted there.
3. sup-2 follows its redirect to gw-1, which redirects it to gw-0.
4. sup-2 is accepted at gw-0 with redirects disabled; gw-0 does not count it yet.
5. sup-0 arrives at gw-0 and is accepted because its local placement state still appears empty.

I reproduced this using the updated GatewayRing and SupervisorSpread implementations with separate spread instances per gateway and the two-hop protocol. Since established hosts do not rebalance, the skew persists after the records catch up. Please immediately account for every accepted host at its actual destination, including hosts that disable redirects, and add coverage with separate placement state per gateway. The current spread tests all exercise one instance.

[FrostGod]

Thanks, reproduced it., : every host a replica keeps now counts there, including hosts that refuse redirects. replicas_count_every_host_they_keep replays your schedule with separate placement state per gateway: 2:1:0 before, 1:1:1 now.

[FrostGod]

/ok to test 9e93a2c

[FrostGod]

/ok to test fed4dae