Repository navigation
Conversation
A supervisor wait recovers from one boundary transport loss, but after that recovery a later loss on the replacement connection is reported as "boundary lost during wait" without any reattach attempt, once the first outage's 30s recovery window has passed. Add a regression test that holds a wait open across two transport aborts more than CONNECT_RETRY_TIMEOUT apart and expects the wait to reattach both times and return the exit status. It fails on this commit; the following commit fixes it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
call_wait_stream opened its 30s recovery window at the first transport failure and never reset it. A wait lasts as long as the workload, so after one successful recovery any later transport loss found the window long expired and returned Unavailable without attempting a reattach. BoundaryProcess::wait() then reported "boundary lost during wait" and the supervisor exited, even though the boundary was still reachable. accept_tcp shares the same path and stopped the proxy the same way. Key the window by connection generation. A failure on a newer generation means a recovery replaced the connection, so it starts a fresh window. Repeated failures on one generation still share a single bounded window, so a live connection whose streams keep failing does not retry forever. This makes wait_survives_a_second_transport_loss_after_recovery pass, and adds unit tests for the per-generation window itself. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mr-miles
force-pushed
the
fix/wait-recovery-window-reset
branch
from
November 12, 2026 20:27
ad53ac8 to
d7b9635
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A supervisor
wait()survived only the first boundary transport loss in a sandbox's lifetime.call_wait_streamopened its 30s recovery window at the first failure and never reset it. After one successful reattach, any later loss found the window already expired and returned without trying to reattach. The supervisor then exited withboundary lost during wait. This PR gives each outage its own window.Related Issue
Closes #5525
Changes
begin_recovery_windowwithWaitRecoveryWindow, which tracks the recovery deadline per connection generation. A failure on a newer generation (a reattach replaced the connection) starts a freshCONNECT_RETRY_TIMEOUTwindow.recover_after_unavailablereturnsOkwithout reconnecting when the probe finds the connection live, and resetting on everyOkwould let a live connection whose streams keep failing retry forever.accept_tcpuses the samecall_wait_streampath, so a second transport loss no longer permanently stops the network proxy either.wait_recovery_window_begins_at_transport_failuretest. Add tests for:Release Notes
Long-running sandboxes no longer exit when the supervisor loses its connection to the sandbox a second time. Each connection loss now gets a full reconnect window.
Testing
cargo fmt --checkcargo test -p openshell-sandbox-backend --libcargo clippy -p openshell-sandbox-backend --all-targetsChecklist