test(server): cover mid-run ServerSet listener failure #119

2026-08-19T18:09:12Z

Lochnair commented

2026-08-19 18:09:12 +00:00

(Migrated from github.com)

Summary

ServerSet already had coverage for construction/bind-time failure
(a_listener_that_cannot_bind_leaves_the_whole_set_unbound) and for
clean shared shutdown, but nothing exercised the other fatal path: every
listener binds and starts serving, and only then does one of them hit an
internal, unrecoverable error. This PR adds exactly that test. No
production behavior changed except a small refactor in runtime.rs to
open a narrow, test-only construction seam — read on for why.

Failure-injection design

ServerCore already has a deliberately private seam for exactly this
kind of thing: TokenSource, which its own unit tests substitute with a
scripted source to drive token-allocation failure deterministically
(ScriptedTokens::failing() in tests/tokens.rs,
a_failing_random_source_is_reported_and_creates_nothing). This PR
reuses that same mechanism at the ServerSet level instead of inventing
new fault-injection machinery.

crates/irtt-server/src/set.rs adds a FailAfterFirstToken token
source: it hands out one real token successfully, then fails every
later draw with ServerError::RandomSource.
To attach it to a real, socket-backed Server (rather than driving
ServerCore directly, which wouldn't touch ServerSet's supervision
at all), crates/irtt-server/src/runtime.rs factors Server::from_socket
into a private Server::with_core and adds a #[cfg(test)] pub(crate) fn from_socket_with_core(socket, core). Production
from_socket is unchanged in behavior — it just calls with_core
with a freshly built ServerCore::new(config), exactly as before.
This is the one narrow production-adjacent hook the task allows for:
no other legitimate/observable route reaches an internal ServerError
deterministically through the public Server/ServerSet API, since
nothing in that API can inject a broken RNG or a failing encoder.

Why this is deterministic and narrow:

No sleeps, no OS-level socket sabotage, no unsafe code, no reliance on
scheduling order.
The listener with the scripted source answers its first open
exactly like a normal listener would (proving it is bound and actively
serving, not merely constructed) — that first successful exchange,
awaited over real loopback UDP with the production encoders, is the
startup/readiness synchronization the test relies on instead of a
sleep.
Only the second open against that same listener draws the second,
failing token, deterministically turning into
ServerError::RandomSource deep inside ServerCore::allocate_token,
which propagates through handle_open → process →
handle_received_datagram → Server::run → the listener's task in
ServerSet::run.

What it proves

The new test,
a_mid_run_listener_failure_shuts_down_its_siblings_and_fails_the_set
in crates/irtt-server/src/set.rs, builds a two-listener ServerSet
(one wired with the failing-after-first-token core, one an ordinary
listener) directly from real bound sockets, and asserts:

both listeners answered a first real open before anything failed
(startup/readiness, not a race);
the injected fatal failure occurs in exactly the one scripted
listener;
ServerSet::run returns
Err(ServerSetError::ListenerRun { addr, source }) naming that exact
listener, with source matching
ServerRuntimeError::Core(ServerError::RandomSource { .. });
the healthy sibling does not linger — both its socket and the failing
listener's socket are provably released (rebindable) after run
returns, which only happens once ServerSet::run's internal
JoinSet has drained both tasks;
no external shutdown is ever sent — the future handed to run is
std::future::pending(), which never resolves, so whatever stopped
the set came entirely from the injected failure.

Result

Behavior was already correct: ServerSet::run's existing fan-out/join
logic (in crates/irtt-server/src/set.rs) handled this exactly as
documented — no bug found, no production behavior changed beyond the
minimal constructor refactor described above.

Verification

cargo fmt --all -- --check
cargo check --workspace --all-targets --all-features --locked
cargo test --workspace --all-targets --all-features --locked
cargo clippy --workspace --all-targets --all-features --locked -- -D warnings
git diff --check

All pass.

Note: the local commit is unsigned. The 1Password ssh-agent used for
commit signing in this repo was unavailable in this unattended session
(1Password: failed to fill whole buffer), so the commit was made with
--no-gpg-sign as the documented fallback. Please sign/re-sign as
needed.

## Summary `ServerSet` already had coverage for construction/bind-time failure (`a_listener_that_cannot_bind_leaves_the_whole_set_unbound`) and for clean shared shutdown, but nothing exercised the other fatal path: every listener binds and starts serving, and only then does one of them hit an internal, unrecoverable error. This PR adds exactly that test. No production behavior changed except a small refactor in `runtime.rs` to open a narrow, test-only construction seam — read on for why. ## Failure-injection design `ServerCore` already has a deliberately private seam for exactly this kind of thing: `TokenSource`, which its own unit tests substitute with a scripted source to drive token-allocation failure deterministically (`ScriptedTokens::failing()` in `tests/tokens.rs`, `a_failing_random_source_is_reported_and_creates_nothing`). This PR reuses that same mechanism at the `ServerSet` level instead of inventing new fault-injection machinery. - `crates/irtt-server/src/set.rs` adds a `FailAfterFirstToken` token source: it hands out one real token successfully, then fails every later draw with `ServerError::RandomSource`. - To attach it to a real, socket-backed `Server` (rather than driving `ServerCore` directly, which wouldn't touch `ServerSet`'s supervision at all), `crates/irtt-server/src/runtime.rs` factors `Server::from_socket` into a private `Server::with_core` and adds a `#[cfg(test)] pub(crate) fn from_socket_with_core(socket, core)`. Production `from_socket` is unchanged in behavior — it just calls `with_core` with a freshly built `ServerCore::new(config)`, exactly as before. This is the one narrow production-adjacent hook the task allows for: no other legitimate/observable route reaches an internal `ServerError` deterministically through the public `Server`/`ServerSet` API, since nothing in that API can inject a broken RNG or a failing encoder. Why this is deterministic and narrow: - No sleeps, no OS-level socket sabotage, no unsafe code, no reliance on scheduling order. - The listener with the scripted source answers its **first** open exactly like a normal listener would (proving it is bound and actively serving, not merely constructed) — that first successful exchange, awaited over real loopback UDP with the production encoders, is the startup/readiness synchronization the test relies on instead of a sleep. - Only the **second** open against that same listener draws the second, failing token, deterministically turning into `ServerError::RandomSource` deep inside `ServerCore::allocate_token`, which propagates through `handle_open` → `process` → `handle_received_datagram` → `Server::run` → the listener's task in `ServerSet::run`. ## What it proves The new test, `a_mid_run_listener_failure_shuts_down_its_siblings_and_fails_the_set` in `crates/irtt-server/src/set.rs`, builds a two-listener `ServerSet` (one wired with the failing-after-first-token core, one an ordinary listener) directly from real bound sockets, and asserts: - both listeners answered a first real open before anything failed (startup/readiness, not a race); - the injected fatal failure occurs in exactly the one scripted listener; - `ServerSet::run` returns `Err(ServerSetError::ListenerRun { addr, source })` naming that exact listener, with `source` matching `ServerRuntimeError::Core(ServerError::RandomSource { .. })`; - the healthy sibling does not linger — both its socket and the failing listener's socket are provably released (rebindable) after `run` returns, which only happens once `ServerSet::run`'s internal `JoinSet` has drained both tasks; - no external shutdown is ever sent — the future handed to `run` is `std::future::pending()`, which never resolves, so whatever stopped the set came entirely from the injected failure. ## Result Behavior was already correct: `ServerSet::run`'s existing fan-out/join logic (in `crates/irtt-server/src/set.rs`) handled this exactly as documented — no bug found, no production behavior changed beyond the minimal constructor refactor described above. ## Verification - `cargo fmt --all -- --check` - `cargo check --workspace --all-targets --all-features --locked` - `cargo test --workspace --all-targets --all-features --locked` - `cargo clippy --workspace --all-targets --all-features --locked -- -D warnings` - `git diff --check` All pass. Note: the local commit is **unsigned**. The 1Password ssh-agent used for commit signing in this repo was unavailable in this unattended session (`1Password: failed to fill whole buffer`), so the commit was made with `--no-gpg-sign` as the documented fallback. Please sign/re-sign as needed.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: Lochnair-Labs/irtt-rs#119