feat(client): capture kernel transmit timestamps #104

2026-08-15T08:59:35Z

Lochnair commented

2026-08-15 08:59:35 +00:00

(Migrated from github.com)

Summary

Linux clients best-effort enable TX_SOFTWARE/OPT_ID/OPT_TSONLY after successful Open;
MSG_ERRQUEUE is drained opportunistically with a fixed work bound;
automatic TX timestamp IDs are correlated with retained probes;
optional kernel TX wall timestamps survive pending -> timeout -> late-reply retention;
no measurement consumes the timestamp yet.

Correlation invariant

The client uses the kernel's automatic OPT_ID directly as the probe's wire sequence number, with no separate correlation map.

A failed/nonblocking send never consumes an ID. Linux increments the per-socket counter (sk->sk_tskey) speculatively while building a datagram and explicitly rolls the increment back (atomic_dec, guarded by a local hold_tskey flag) on every error path out of __ip_append_data/__ip6_append_data (net/ipv4/ip_output.c, net/ipv6/ip6_output.c); the assignment itself (_sock_tx_timestamp, include/net/sock.h) is gated on SOF_TIMESTAMPING_OPT_ID. A WouldBlock try_send therefore can never leave a gap in, or otherwise advance, the ID the kernel hands to the next successful submission. Client-side, commit_probe_sent — the only thing that advances next_wire_seq — only runs after a confirmed successful send; existing tests (repeated_would_block_style_preflight_commits_once in machine.rs, and retains_prepared_probe_across_readiness_false_positives_and_cancellation in async_client/tests.rs, which already asserts EchoSent { seq: 0, .. } after an injected WouldBlock) prove the client never advances past a failed attempt.
First ID after post-Open enable. wire_seq starts at 0 on every successful Open (SessionMachine::commit_open builds a fresh session with next_wire_seq: 0) and TX timestamping is enabled only after that same successful Open, before any probe is sent — so both counters' zero points line up exactly. Linux smoke tests prove sends made before the upgrade never occupy the later namespace, and the first send after upgrade reports id 0.
One socket, one session, no reopen. SessionMachine's state machine only reaches Open from Connected, and neither commit_open nor commit_local_close route back to Connected — a Client/AsyncClient cannot reopen a second session on the same socket, so there's no second epoch to collide with a stale kernel ID.
Wrap behavior matches. wire_seq wraps via u32::wrapping_add; the kernel counter is a plain u32 atomic. A dedicated test (kernel_tx_timestamp_ids_wrap_like_wire_seq) proves correlation at the u32::MAX -> 0 boundary.
Close is harmless. Close sends one datagram after TX timestamping is already enabled, generating one unmatched TX timestamp with no corresponding wire_seq (Close doesn't advance next_wire_seq). Because the session is terminal immediately after, nothing ever looks it up; the "unknown ID -> ignore" rule discards it safely.

Error queue

error_queue::classify uses origin (SO_EE_ORIGIN_TIMESTAMPING or not) as the sole discriminator between timing metadata and a real socket error. A timestamping-origin record with an unexpected completion kind or missing/invalid timestamp becomes MalformedOrUnsupportedTimestamp (dormant metadata, never fatal) instead of the previous, over-aggressive SocketError. Only a non-timestamping-origin record becomes SocketError.
drain_tx_timestamps reads one record at a time, no heap allocation, up to MAX_TX_TIMESTAMP_RECORDS_PER_DRAIN (32) per call — a starvation guard, not a queue-capacity promise, since one TX timestamp is generated per successful probe and drains happen at several choke points.
No background task; the Tokio adapter uses the same raw nonblocking recvmsg(MSG_ERRQUEUE | MSG_DONTWAIT) rather than try_io/poll_recv_ready, since #103 established that error-queue readiness is not ordinary read readiness and driving it through the reactor risks disturbing the normal Echo receive path.
A genuine SocketError record stops the drain and is surfaced through the existing ClientError::Socket path.

Capability/fallback

TX setup failure is fully non-fatal: try_enable_tx_timestamping best-effort restores the original RX-only flags on failure and returns false; the client tracks that in a small adapter-owned bool (not SessionMachine state) so unnecessary drains are skipped.
Timestamp absence/mismatch never fails a probe.
Non-Linux and non-ancillary builds are byte-for-byte unchanged; no new send path, no new syscall.

State

PendingProbe gains kernel_tx_timestamp: Option<SystemTime> alongside the authoritative sent_at. SessionMachine::record_kernel_tx_timestamp attaches an observed timestamp to a still-pending or already-timed-out probe (first valid observation wins on duplicates), never resurrects a completed/evicted/unknown probe, and never touches sent_at, timeout/loss/reply state, or any counter. No new unbounded state — it's one Option riding along inside the existing bounded PendingMap/TimedOutMap.

Measurements

No OWD / RTT / IPDV behavior changes in this PR. Kernel TX timestamp is captured and correlated only; kernel_tx_timestamp_presence_does_not_change_rtt_or_owd and measurable_late_reply_retains_the_kernel_tx_timestamp assert this directly. sent_at remains the sole input to RTT, timeout, and all one-way delay/IPDV computations.

Scope

No: preferred-send-wall activation, F-04 send-timestamp movement, public kernel_tx field/event, sendmsg/SCM_TS_OPT_ID, hardware timing, unsafe, background task.

Tests

automatic-ID enable-after-prior-sends (rx_only_sends_before_tx_enable_do_not_consume_the_later_tx_id_namespace)
Linux error-queue integration and classification re-audit (error_queue.rs)
correlation / out-of-order / duplicate / wrap / timeout / late-retention / eviction / close (session/machine.rs)
adapter-level enable+drain end-to-end (receive/linux.rs)
dormant-measurement regression (kernel_tx_timestamp_presence_does_not_change_rtt_or_owd)

Verification

cargo test -p irtt-client --all-features --locked
cargo test -p irtt-stats --all-features --locked
cargo test -p irtt-cli --all-features --locked
cargo fmt --all -- --check
cargo check --workspace --all-targets --all-features --locked
cargo test --workspace --all-targets --all-features --locked
cargo clippy --workspace --all-targets --all-features --locked -- -D warnings
RUSTDOCFLAGS="-D warnings" cargo doc --workspace --all-features --no-deps --locked
cargo check -p irtt-cli --no-default-features --features {client,tui,server} --all-targets --locked
git diff --check
Cross-target cargo check -p irtt-client --all-features: x86_64-unknown-linux-gnu, aarch64-unknown-linux-musl, x86_64-unknown-freebsd, x86_64-pc-windows-gnu, i686-unknown-linux-gnu
Linux-only runtime tests (smoke tests, error-queue integration) run in rust:1-bookworm via Docker, since the dev host is macOS: 347/347 passed.

## Summary - Linux clients best-effort enable TX_SOFTWARE/OPT_ID/OPT_TSONLY after successful Open; - MSG_ERRQUEUE is drained opportunistically with a fixed work bound; - automatic TX timestamp IDs are correlated with retained probes; - optional kernel TX wall timestamps survive pending -> timeout -> late-reply retention; - no measurement consumes the timestamp yet. ## Correlation invariant The client uses the kernel's automatic `OPT_ID` directly as the probe's wire sequence number, with no separate correlation map. - **A failed/nonblocking send never consumes an ID.** Linux increments the per-socket counter (`sk->sk_tskey`) speculatively while building a datagram and explicitly rolls the increment back (`atomic_dec`, guarded by a local `hold_tskey` flag) on every error path out of `__ip_append_data`/`__ip6_append_data` (`net/ipv4/ip_output.c`, `net/ipv6/ip6_output.c`); the assignment itself (`_sock_tx_timestamp`, `include/net/sock.h`) is gated on `SOF_TIMESTAMPING_OPT_ID`. A `WouldBlock` `try_send` therefore can never leave a gap in, or otherwise advance, the ID the kernel hands to the next *successful* submission. Client-side, `commit_probe_sent` — the only thing that advances `next_wire_seq` — only runs after a confirmed successful send; existing tests (`repeated_would_block_style_preflight_commits_once` in `machine.rs`, and `retains_prepared_probe_across_readiness_false_positives_and_cancellation` in `async_client/tests.rs`, which already asserts `EchoSent { seq: 0, .. }` after an injected `WouldBlock`) prove the client never advances past a failed attempt. - **First ID after post-Open enable.** `wire_seq` starts at 0 on every successful Open (`SessionMachine::commit_open` builds a fresh session with `next_wire_seq: 0`) and TX timestamping is enabled only after that same successful Open, before any probe is sent — so both counters' zero points line up exactly. Linux smoke tests prove sends made before the upgrade never occupy the later namespace, and the first send after upgrade reports id 0. - **One socket, one session, no reopen.** `SessionMachine`'s state machine only reaches `Open` from `Connected`, and neither `commit_open` nor `commit_local_close` route back to `Connected` — a `Client`/`AsyncClient` cannot reopen a second session on the same socket, so there's no second epoch to collide with a stale kernel ID. - **Wrap behavior matches.** `wire_seq` wraps via `u32::wrapping_add`; the kernel counter is a plain `u32` atomic. A dedicated test (`kernel_tx_timestamp_ids_wrap_like_wire_seq`) proves correlation at the `u32::MAX -> 0` boundary. - **Close is harmless.** `Close` sends one datagram after TX timestamping is already enabled, generating one unmatched TX timestamp with no corresponding `wire_seq` (`Close` doesn't advance `next_wire_seq`). Because the session is terminal immediately after, nothing ever looks it up; the "unknown ID -> ignore" rule discards it safely. ## Error queue - `error_queue::classify` uses *origin* (`SO_EE_ORIGIN_TIMESTAMPING` or not) as the sole discriminator between timing metadata and a real socket error. A timestamping-origin record with an unexpected completion kind or missing/invalid timestamp becomes `MalformedOrUnsupportedTimestamp` (dormant metadata, never fatal) instead of the previous, over-aggressive `SocketError`. Only a non-timestamping-origin record becomes `SocketError`. - `drain_tx_timestamps` reads one record at a time, no heap allocation, up to `MAX_TX_TIMESTAMP_RECORDS_PER_DRAIN` (32) per call — a starvation guard, not a queue-capacity promise, since one TX timestamp is generated per successful probe and drains happen at several choke points. - No background task; the Tokio adapter uses the same raw nonblocking `recvmsg(MSG_ERRQUEUE | MSG_DONTWAIT)` rather than `try_io`/`poll_recv_ready`, since #103 established that error-queue readiness is not ordinary read readiness and driving it through the reactor risks disturbing the normal Echo receive path. - A genuine `SocketError` record stops the drain and is surfaced through the existing `ClientError::Socket` path. ## Capability/fallback - TX setup failure is fully non-fatal: `try_enable_tx_timestamping` best-effort restores the original RX-only flags on failure and returns `false`; the client tracks that in a small adapter-owned `bool` (not `SessionMachine` state) so unnecessary drains are skipped. - Timestamp absence/mismatch never fails a probe. - Non-Linux and non-`ancillary` builds are byte-for-byte unchanged; no new send path, no new syscall. ## State `PendingProbe` gains `kernel_tx_timestamp: Option<SystemTime>` alongside the authoritative `sent_at`. `SessionMachine::record_kernel_tx_timestamp` attaches an observed timestamp to a still-pending or already-timed-out probe (first valid observation wins on duplicates), never resurrects a completed/evicted/unknown probe, and never touches `sent_at`, timeout/loss/reply state, or any counter. No new unbounded state — it's one `Option` riding along inside the existing bounded `PendingMap`/`TimedOutMap`. ## Measurements **No OWD / RTT / IPDV behavior changes in this PR.** Kernel TX timestamp is captured and correlated only; `kernel_tx_timestamp_presence_does_not_change_rtt_or_owd` and `measurable_late_reply_retains_the_kernel_tx_timestamp` assert this directly. `sent_at` remains the sole input to RTT, timeout, and all one-way delay/IPDV computations. ## Scope No: preferred-send-wall activation, F-04 send-timestamp movement, public `kernel_tx` field/event, `sendmsg`/`SCM_TS_OPT_ID`, hardware timing, `unsafe`, background task. ## Tests - automatic-ID enable-after-prior-sends (`rx_only_sends_before_tx_enable_do_not_consume_the_later_tx_id_namespace`) - Linux error-queue integration and classification re-audit (`error_queue.rs`) - correlation / out-of-order / duplicate / wrap / timeout / late-retention / eviction / close (`session/machine.rs`) - adapter-level enable+drain end-to-end (`receive/linux.rs`) - dormant-measurement regression (`kernel_tx_timestamp_presence_does_not_change_rtt_or_owd`) ## Verification - `cargo test -p irtt-client --all-features --locked` - `cargo test -p irtt-stats --all-features --locked` - `cargo test -p irtt-cli --all-features --locked` - `cargo fmt --all -- --check` - `cargo check --workspace --all-targets --all-features --locked` - `cargo test --workspace --all-targets --all-features --locked` - `cargo clippy --workspace --all-targets --all-features --locked -- -D warnings` - `RUSTDOCFLAGS="-D warnings" cargo doc --workspace --all-features --no-deps --locked` - `cargo check -p irtt-cli --no-default-features --features {client,tui,server} --all-targets --locked` - `git diff --check` - Cross-target `cargo check -p irtt-client --all-features`: `x86_64-unknown-linux-gnu`, `aarch64-unknown-linux-musl`, `x86_64-unknown-freebsd`, `x86_64-pc-windows-gnu`, `i686-unknown-linux-gnu` - Linux-only runtime tests (smoke tests, error-queue integration) run in `rust:1-bookworm` via Docker, since the dev host is macOS: 347/347 passed.

👍 1

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: Lochnair-Labs/irtt-rs#104