fix(io): retry interrupted receives #111

2026-08-16T17:24:09Z

Lochnair commented

2026-08-16 17:24:09 +00:00

(Migrated from github.com)

Summary

Real signal delivery on Linux (surfaced by an x86_64-musl workspace test run) can interrupt blocking and nonblocking recv/recvmsg calls with EINTR (io::ErrorKind::Interrupted). The failing case was irtt-cli's fake-server test helpers, which called recv_from(...).unwrap(); investigating that panic surfaced several production receive paths that would also treat Interrupted as a fatal error rather than a transient interruption.

Production paths affected:

Client::open's blocking receive loop (crates/irtt-client/src/client.rs) — returned ClientError::Socket on Interrupted instead of retrying within the current open attempt.
Client::recv_once / recv_available's blocking receive (same file) — same fatal-error behavior instead of retrying within the receive's timeout budget.
AsyncClient::poll_open and poll_recv (crates/irtt-client/src/async_client.rs) — the Linux ancillary nonblocking recvmsg path failed the receive instead of retrying through the existing readiness/work-budget machinery.
The server's ancillary receive loop (crates/irtt-server/src/socket_io/ancillary.rs) — an interrupted nonblocking recvmsg could terminate the listener.
The client's MSG_ERRQUEUE TX-timestamp drain (crates/irtt-client/src/receive/linux/error_queue.rs) — turned an interrupted read of optional metadata into a fatal ClientError::Socket.

Test-only hazard: crates/irtt-cli/tests/client_multi_target.rs and crates/irtt-client/tests/support/mod.rs fake-server receive helpers either panicked on any error (recv_from(...).unwrap()) or silently misclassified every error, including Interrupted, as "peer went quiet."

Correctness

Interrupted means the operation did not produce its requested result — it is not a malformed packet, protocol failure, peer failure, real socket/network error, or timeout, and it must be invisible to protocol/session state.

Blocking Open already computes one absolute per-attempt deadline before its receive loop and recomputes remaining/re-applies SO_RCVTIMEO on every loop iteration regardless of why it looped. Adding Interrupted to the same retry arm as WouldBlock/TimedOut means an interruption resumes inside the same attempt, with no retransmit and no deadline reset — it just reuses machinery that was already there.
Blocking recv_once/recv_available now compute one absolute deadline per logical receive when a timeout is configured (None keeps blocking indefinitely after an interruption, matching its no-timeout contract), and restore the socket's configured read timeout before returning so a narrowed timeout from one interrupted receive can't leak into the next call inside a recv_available budget.
AsyncClient/Tokio: poll_open's retry loop was already bounded by OPEN_POLL_WORK_BUDGET, so adding Interrupted to its retry arm was sufficient. poll_recv had no such bound — and needed one, because try_io only clears Tokio's readiness state on WouldBlock, not on Interrupted, so a sustained signal delivery rate could otherwise retry in a tight loop without a genuine new readiness edge. It now shares the same RECV_POLL_WORK_BUDGET-bounded, wake-and-yield pattern poll_open already used.
Server ancillary receive: same readiness-clearing asymmetry applies to socket.readable().await after try_io. The loop now retries Interrupted like WouldBlock, and yields via tokio::task::yield_now() after a bounded run of consecutive interruptions so a signal storm can't busy-loop the listener task — while never treating the interruption as a reason to terminate the listener.
TX timestamp drain: capture is explicitly best-effort optional metadata, not a network operation. An interrupted MSG_ERRQUEUE read is now treated as no record obtained on this drain (Ok(None), matching the existing EWOULDBLOCK case) — the bounded caller naturally revisits the queue on the next natural drain point rather than spinning to retry optional metadata. Genuine SocketError records are unaffected.

No packet/session counters can move from an Interrupted receive alone, no protocol/timeout classification changed, and no unconditional/unbounded inner retry loops were introduced.

Tests

Deterministic, test-only injection hooks (mirroring this codebase's existing send-injection test hooks) simulate Interrupted without relying on flaky real signal timing:

open_survives_interrupted_receive_within_same_attempt / open_interrupted_receive_does_not_extend_the_attempt_deadline (client/tests/open.rs) — Open completes despite repeated interruptions with exactly one request sent (no retransmit), and repeated interruptions against a silent server still hit OpenTimeout within the configured attempt window.
interrupted_receive_is_retried_and_the_reply_is_still_classified / interrupted_receive_does_not_extend_the_configured_timeout (client/tests/probes_replies/interrupted_receive.rs) — recv_once classifies a reply normally despite interruptions, and a receive with no reply pending still times out within its configured budget.
open_retries_interrupted_syscall_within_the_same_attempt / receive_retries_interrupted_syscall_and_shares_packet_classification (async_client/tests.rs) — same two invariants for AsyncClient.

Also fixed the originally-observed test-only hazard: client_multi_target.rs's and tests/support/mod.rs's fake-server recv_from helpers now retry Interrupted instead of panicking or misclassifying it as "peer quiet."

Verification run:

cargo fmt --all -- --check
cargo check --workspace --all-targets --all-features --locked
cargo test --workspace --all-targets --all-features --locked (all crates pass)
cargo clippy --workspace --all-targets --all-features --locked -- -D warnings
RUSTDOCFLAGS="-D warnings" cargo doc --workspace --all-features --no-deps --locked
cargo check/test -p irtt-client --no-default-features --locked
cargo check/test -p irtt-client --all-features --locked
cargo test -p irtt-server --all-features --locked
git diff --check
Real Linux + musl libc: cross-compiled --target aarch64-unknown-linux-musl and ran every resulting test binary (26 binaries, including the Linux-ancillary-only receive/error-queue/socket_io unit tests that only compile under target_os = "linux") natively inside an OrbStack Ubuntu (arm64) VM — all passed, 0 failures. (x86_64-unknown-linux-musl was unavailable in this environment without a QEMU user-mode emulator; aarch64-musl on a real Linux kernel exercises the identical recvmsg/EINTR/Tokio-readiness code paths this PR touches, so it stands in for the originally observed x86_64-musl run.)

Scope

No protocol, session, sequence, or timeout-classification behavior changes.
No new dependencies.
Send-side EINTR handling is unchanged and out of scope for this PR — a few nonblocking-send WouldBlock-only match arms (e.g. the server's send_from_source, AsyncClient::poll_send_probe/poll_close) could in principle see the same asymmetry, but nothing in this investigation showed a production send observing Interrupted, so it's left as a candidate follow-up rather than folded into this receive-side fix.
A few structurally-identical .unwrap() fake-server receive helpers exist elsewhere in the workspace (irtt-client/src/managed/{blocking_tests,tests}.rs, client/tests/support.rs, client/tests/probes_replies/{timeouts,reply_metadata}.rs) but were left alone to keep this a narrow fix rather than a workspace-wide test-helper cleanup; noted here as a known, lower-risk, single-call-site (not looped) test-only hazard rather than silently ignored.

🤖 Generated with Claude Code

## Summary Real signal delivery on Linux (surfaced by an x86_64-musl workspace test run) can interrupt blocking and nonblocking `recv`/`recvmsg` calls with `EINTR` (`io::ErrorKind::Interrupted`). The failing case was `irtt-cli`'s fake-server test helpers, which called `recv_from(...).unwrap()`; investigating that panic surfaced several production receive paths that would also treat `Interrupted` as a fatal error rather than a transient interruption. Production paths affected: - `Client::open`'s blocking receive loop (`crates/irtt-client/src/client.rs`) — returned `ClientError::Socket` on `Interrupted` instead of retrying within the current open attempt. - `Client::recv_once` / `recv_available`'s blocking receive (same file) — same fatal-error behavior instead of retrying within the receive's timeout budget. - `AsyncClient::poll_open` and `poll_recv` (`crates/irtt-client/src/async_client.rs`) — the Linux ancillary nonblocking `recvmsg` path failed the receive instead of retrying through the existing readiness/work-budget machinery. - The server's ancillary receive loop (`crates/irtt-server/src/socket_io/ancillary.rs`) — an interrupted nonblocking `recvmsg` could terminate the listener. - The client's `MSG_ERRQUEUE` TX-timestamp drain (`crates/irtt-client/src/receive/linux/error_queue.rs`) — turned an interrupted read of optional metadata into a fatal `ClientError::Socket`. Test-only hazard: `crates/irtt-cli/tests/client_multi_target.rs` and `crates/irtt-client/tests/support/mod.rs` fake-server receive helpers either panicked on any error (`recv_from(...).unwrap()`) or silently misclassified every error, including `Interrupted`, as "peer went quiet." ## Correctness `Interrupted` means the operation did not produce its requested result — it is not a malformed packet, protocol failure, peer failure, real socket/network error, or timeout, and it must be invisible to protocol/session state. - **Blocking Open** already computes one absolute per-attempt deadline before its receive loop and recomputes `remaining`/re-applies `SO_RCVTIMEO` on every loop iteration regardless of why it looped. Adding `Interrupted` to the same retry arm as `WouldBlock`/`TimedOut` means an interruption resumes inside the same attempt, with no retransmit and no deadline reset — it just reuses machinery that was already there. - **Blocking `recv_once`/`recv_available`** now compute one absolute deadline per logical receive when a timeout is configured (`None` keeps blocking indefinitely after an interruption, matching its no-timeout contract), and restore the socket's configured read timeout before returning so a narrowed timeout from one interrupted receive can't leak into the next call inside a `recv_available` budget. - **AsyncClient/Tokio**: `poll_open`'s retry loop was already bounded by `OPEN_POLL_WORK_BUDGET`, so adding `Interrupted` to its retry arm was sufficient. `poll_recv` had no such bound — and needed one, because `try_io` only clears Tokio's readiness state on `WouldBlock`, not on `Interrupted`, so a sustained signal delivery rate could otherwise retry in a tight loop without a genuine new readiness edge. It now shares the same `RECV_POLL_WORK_BUDGET`-bounded, wake-and-yield pattern `poll_open` already used. - **Server ancillary receive**: same readiness-clearing asymmetry applies to `socket.readable().await` after `try_io`. The loop now retries `Interrupted` like `WouldBlock`, and yields via `tokio::task::yield_now()` after a bounded run of consecutive interruptions so a signal storm can't busy-loop the listener task — while never treating the interruption as a reason to terminate the listener. - **TX timestamp drain**: capture is explicitly best-effort optional metadata, not a network operation. An interrupted `MSG_ERRQUEUE` read is now treated as no record obtained on this drain (`Ok(None)`, matching the existing `EWOULDBLOCK` case) — the bounded caller naturally revisits the queue on the next natural drain point rather than spinning to retry optional metadata. Genuine `SocketError` records are unaffected. No packet/session counters can move from an `Interrupted` receive alone, no protocol/timeout classification changed, and no unconditional/unbounded inner retry loops were introduced. ## Tests Deterministic, test-only injection hooks (mirroring this codebase's existing send-injection test hooks) simulate `Interrupted` without relying on flaky real signal timing: - `open_survives_interrupted_receive_within_same_attempt` / `open_interrupted_receive_does_not_extend_the_attempt_deadline` (`client/tests/open.rs`) — Open completes despite repeated interruptions with exactly one request sent (no retransmit), and repeated interruptions against a silent server still hit `OpenTimeout` within the configured attempt window. - `interrupted_receive_is_retried_and_the_reply_is_still_classified` / `interrupted_receive_does_not_extend_the_configured_timeout` (`client/tests/probes_replies/interrupted_receive.rs`) — `recv_once` classifies a reply normally despite interruptions, and a receive with no reply pending still times out within its configured budget. - `open_retries_interrupted_syscall_within_the_same_attempt` / `receive_retries_interrupted_syscall_and_shares_packet_classification` (`async_client/tests.rs`) — same two invariants for `AsyncClient`. Also fixed the originally-observed test-only hazard: `client_multi_target.rs`'s and `tests/support/mod.rs`'s fake-server `recv_from` helpers now retry `Interrupted` instead of panicking or misclassifying it as "peer quiet." Verification run: - `cargo fmt --all -- --check` - `cargo check --workspace --all-targets --all-features --locked` - `cargo test --workspace --all-targets --all-features --locked` (all crates pass) - `cargo clippy --workspace --all-targets --all-features --locked -- -D warnings` - `RUSTDOCFLAGS="-D warnings" cargo doc --workspace --all-features --no-deps --locked` - `cargo check`/`test -p irtt-client --no-default-features --locked` - `cargo check`/`test -p irtt-client --all-features --locked` - `cargo test -p irtt-server --all-features --locked` - `git diff --check` - Real Linux + musl libc: cross-compiled `--target aarch64-unknown-linux-musl` and ran every resulting test binary (26 binaries, including the Linux-ancillary-only receive/error-queue/socket_io unit tests that only compile under `target_os = "linux"`) natively inside an OrbStack Ubuntu (arm64) VM — all passed, 0 failures. (x86_64-unknown-linux-musl was unavailable in this environment without a QEMU user-mode emulator; aarch64-musl on a real Linux kernel exercises the identical `recvmsg`/`EINTR`/Tokio-readiness code paths this PR touches, so it stands in for the originally observed x86_64-musl run.) ## Scope - No protocol, session, sequence, or timeout-classification behavior changes. - No new dependencies. - Send-side `EINTR` handling is unchanged and out of scope for this PR — a few nonblocking-send `WouldBlock`-only match arms (e.g. the server's `send_from_source`, `AsyncClient::poll_send_probe`/`poll_close`) could in principle see the same asymmetry, but nothing in this investigation showed a production send observing `Interrupted`, so it's left as a candidate follow-up rather than folded into this receive-side fix. - A few structurally-identical `.unwrap()` fake-server receive helpers exist elsewhere in the workspace (`irtt-client/src/managed/{blocking_tests,tests}.rs`, `client/tests/support.rs`, `client/tests/probes_replies/{timeouts,reply_metadata}.rs`) but were left alone to keep this a narrow fix rather than a workspace-wide test-helper cleanup; noted here as a known, lower-risk, single-call-site (not looped) test-only hazard rather than silently ignored. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

chatgpt-codex-connector[bot] (Migrated from github.com) reviewed

2026-08-16 17:28:56 +00:00

chatgpt-codex-connector[bot] (Migrated from github.com) left a comment

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ca2389174f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

### 💡 Codex Review Here are some automated review suggestions for this pull request. **Reviewed commit:** `ca2389174f` <details> <summary>ℹ️ About Codex in GitHub</summary> [Your team has set up Codex to review pull requests in this repo](https://chatgpt.com/codex/cloud/settings/general). Reviews are triggered when you - Open a pull request for review - Mark a draft as ready - Comment "@codex review". If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". </details>

crates/irtt-cli/tests/client_multi_target.rs

						
				@@ -1177,16 +1177,66 @@ fn start_open_failure_server_inner(

				    FakeServer { addr, done }

chatgpt-codex-connector[bot] (Migrated from github.com) commented

2026-08-16 17:28:56 +00:00

Preserve the timeout deadline across test-server interrupts

When this helper runs under repeated signals spaced closer than its configured 500 ms socket timeout, every Interrupted result starts a new recv_from with the full SO_RCVTIMEO, so the loop has no absolute deadline. start_interruptible_echo_server subsequently joins this thread synchronously, meaning sustained process-group signals can hang the test suite instead of letting the fake peer become quiet. Keep one absolute deadline and apply only the remaining timeout on each retry, as the production blocking receive now does.

AGENTS.md reference: AGENTS.md:L143-L150

Useful? React with 👍 / 👎.

**![P2 Badge](https://img.shields.io/badge/P2-yellow?style=flat) Preserve the timeout deadline across test-server interrupts** When this helper runs under repeated signals spaced closer than its configured 500 ms socket timeout, every `Interrupted` result starts a new `recv_from` with the full `SO_RCVTIMEO`, so the loop has no absolute deadline. `start_interruptible_echo_server` subsequently joins this thread synchronously, meaning sustained process-group signals can hang the test suite instead of letting the fake peer become quiet. Keep one absolute deadline and apply only the remaining timeout on each retry, as the production blocking receive now does. AGENTS.md reference: [AGENTS.md:L143-L150](https://github.com/Lochnair/irtt-rs/blob/ca2389174fd0e103fe645fd3a9f6ee45b3c69429/AGENTS.md#L143-L150) Useful? React with 👍 / 👎.

crates/irtt-client/src/receive/linux/error_queue.rs

chatgpt-codex-connector[bot] (Migrated from github.com) commented

2026-08-16 17:28:56 +00:00

Continue the bounded drain after an interrupted read

When Linux recvmsg(MSG_ERRQUEUE) is interrupted during the final drain immediately before an echo reply is processed, mapping EINTR to None makes drain_tx_timestamps stop as though the queue were empty. The timestamp record remains queued, but processing the reply removes its pending probe, so a later drain cannot associate the record and the upstream one-way-delay measurement permanently falls back to the less accurate userspace timestamp. Treat the interruption as a skipped attempt within the existing 32-iteration work budget rather than as an empty queue.

AGENTS.md reference: crates/irtt-client/AGENTS.md:L77-L86

Useful? React with 👍 / 👎.

**![P2 Badge](https://img.shields.io/badge/P2-yellow?style=flat) Continue the bounded drain after an interrupted read** When Linux `recvmsg(MSG_ERRQUEUE)` is interrupted during the final drain immediately before an echo reply is processed, mapping `EINTR` to `None` makes `drain_tx_timestamps` stop as though the queue were empty. The timestamp record remains queued, but processing the reply removes its pending probe, so a later drain cannot associate the record and the upstream one-way-delay measurement permanently falls back to the less accurate userspace timestamp. Treat the interruption as a skipped attempt within the existing 32-iteration work budget rather than as an empty queue. AGENTS.md reference: [crates/irtt-client/AGENTS.md:L77-L86](https://github.com/Lochnair/irtt-rs/blob/ca2389174fd0e103fe645fd3a9f6ee45b3c69429/crates/irtt-client/AGENTS.md#L77-L86) Useful? React with 👍 / 👎.

chatgpt-codex-connector[bot] commented

2026-08-16 17:32:45 +00:00

(Migrated from github.com)

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, you can upgrade your account or add credits to your account and enable them for code reviews in your settings.

You have reached your Codex usage limits for code reviews. You can see your limits in the [Codex usage dashboard](https://chatgpt.com/codex/cloud/settings/usage). To continue using code reviews, you can upgrade your account or add credits to your account and enable them for code reviews in your [settings](https://chatgpt.com/codex/cloud/settings/code-review).

Lochnair (Migrated from github.com) reviewed

2026-08-16 17:32:52 +00:00

crates/irtt-client/src/receive/linux/error_queue.rs

Lochnair (Migrated from github.com) commented

2026-08-16 17:32:52 +00:00

Fixed: an interrupted MSG_ERRQUEUE read now reports ErrorQueueRecord::Ignored instead of None, so it consumes one attempt of the drain's existing per-call work budget and retries the same slot rather than ending the drain early. See a598dbd.

🤖 Addressed by Claude Code

Fixed: an interrupted `MSG_ERRQUEUE` read now reports `ErrorQueueRecord::Ignored` instead of `None`, so it consumes one attempt of the drain's existing per-call work budget and retries the same slot rather than ending the drain early. See a598dbd. _🤖 Addressed by [Claude Code](https://claude.com/claude-code)_

Lochnair (Migrated from github.com) reviewed

2026-08-16 17:32:59 +00:00

crates/irtt-cli/tests/client_multi_target.rs

						
				@@ -1177,16 +1177,66 @@ fn start_open_failure_server_inner(

				    FakeServer { addr, done }

Lochnair (Migrated from github.com) commented

2026-08-16 17:32:59 +00:00

Fixed: recv_request/recv_request_timeout now share a recv_from_retrying_interrupted helper that computes one absolute deadline from the socket's configured read timeout and re-applies only the remaining time on each interrupted retry, restoring the configured timeout before returning. Applied the same fix to the equivalent helpers in irtt-client/tests/support/mod.rs. See a598dbd.

🤖 Addressed by Claude Code

Fixed: `recv_request`/`recv_request_timeout` now share a `recv_from_retrying_interrupted` helper that computes one absolute deadline from the socket's configured read timeout and re-applies only the remaining time on each interrupted retry, restoring the configured timeout before returning. Applied the same fix to the equivalent helpers in `irtt-client/tests/support/mod.rs`. See a598dbd. _🤖 Addressed by [Claude Code](https://claude.com/claude-code)_

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: Lochnair-Labs/irtt-rs#111

				`@@ -1177,16 +1177,66 @@ fn start_open_failure_server_inner(`
				`FakeServer { addr, done }`