Nils Andreas Svee
9e3477e617
On Linux, best-effort upgrade the socket's SO_TIMESTAMPING flags to TX_SOFTWARE | OPT_ID | OPT_TSONLY after a successful Open and before the first Echo probe. Promote the #103 error-queue foundation to production: MSG_ERRQUEUE is drained opportunistically with a fixed per-call budget at several natural choke points (after a probe send, before processing a reply, before evaluating timeouts), and each automatic OPT_ID timestamp is correlated directly with wire_seq — safe because Linux rolls back its speculative sk_tskey increment on every send failure, wire_seq and the kernel counter start together at every Open, and a socket can never be reopened for a second session. The observed kernel TX wall timestamp is retained on PendingProbe (surviving pending -> timed-out retention) as dormant metadata only; RTT, OWD, and IPDV are unchanged and continue to use the userspace send timestamp exclusively. Capability failure is fully non-fatal and RX timestamping is unaffected either way. Non-Linux and non-ancillary builds are unchanged.
109 lines
6.0 KiB
Markdown
109 lines
6.0 KiB
Markdown
# `irtt-client` guidance
|
|
|
|
Repository-wide guidance in the root `AGENTS.md` also applies here.
|
|
|
|
## Linux kernel TX timestamp capture
|
|
|
|
On Linux, with the `ancillary` feature, the client best-effort upgrades its
|
|
socket from RX-only `SO_TIMESTAMPING` to RX+TX immediately after a successful
|
|
Open, before the first Echo probe can be sent:
|
|
|
|
```
|
|
socket created
|
|
-> RX timestamping configured (RX_SOFTWARE | SOFTWARE)
|
|
-> Open attempts/retries (untimestamped)
|
|
-> successful Open
|
|
-> best-effort enable TX timestamping (+ TX_SOFTWARE | OPT_ID | OPT_TSONLY)
|
|
-> first Echo probe
|
|
```
|
|
|
|
`TX_SOFTWARE` asks the kernel for a software send timestamp; `OPT_ID` asks it
|
|
to tag each notification with an automatically incrementing per-socket ID;
|
|
`OPT_TSONLY` means the notification never needs to carry the original
|
|
payload, so a zero-length receive buffer is enough. The upgrade is entirely
|
|
best-effort: if it fails, the client keeps operating on the original RX-only
|
|
configuration (`try_enable_tx_timestamping` restores it explicitly) and
|
|
timing falls back to userspace `sent_at` as it always has. TX capability
|
|
failure is never a public error and is never logged from the library.
|
|
|
|
### Correlation invariant: kernel ID == wire sequence
|
|
|
|
The client uses the kernel's automatic `OPT_ID` directly as the probe's wire
|
|
sequence number — no separate correlation map. This is safe because:
|
|
|
|
- **A failed/nonblocking send never consumes an ID.** Linux increments the
|
|
per-socket ID counter (`sk->sk_tskey`) speculatively while building a
|
|
datagram and rolls the increment back (`atomic_dec`) on every error path
|
|
out of `__ip_append_data`/`__ip6_append_data`
|
|
(`net/ipv4/ip_output.c`, `net/ipv6/ip6_output.c`), guarded by a local
|
|
`hold_tskey` flag. A `WouldBlock` `try_send` therefore can never leave a
|
|
gap in, or otherwise advance, the ID the kernel will assign to the next
|
|
*successful* submission. (`include/net/sock.h`'s `_sock_tx_timestamp`
|
|
performs the same increment for the non-corked path.) Client-side, the
|
|
existing send transaction only calls `commit_probe_sent` — which is the
|
|
only thing that advances `next_wire_seq` — after a successful socket send;
|
|
a `WouldBlock` result never reaches it.
|
|
- **The two counters start together and advance together.** `wire_seq`
|
|
starts at 0 on every successful Open (`SessionMachine::commit_open` builds
|
|
a fresh `ActiveSession` with `next_wire_seq: 0`) and advances by exactly 1
|
|
only via `commit_probe_sent`, which only runs after a confirmed successful
|
|
send — the same condition under which the kernel's counter advances by 1.
|
|
TX timestamping is enabled only after that same successful Open and before
|
|
any probe is sent, so both counters' zero points line up exactly.
|
|
- **One socket, one session, no reopen.** `SessionMachine`'s state machine
|
|
only reaches `Open` from `Connected`, and `commit_open`/`commit_local_close`
|
|
never route back to `Connected`. A `Client`/`AsyncClient` cannot reopen a
|
|
second session on the same socket, so there is no second epoch of wire
|
|
sequence numbers to collide with a stale kernel ID.
|
|
- **The one extra send (Close) is harmless.** `Close` sends one datagram
|
|
after TX timestamping is already enabled, generating one TX timestamp
|
|
record with no corresponding `wire_seq` (`Close` does not advance
|
|
`next_wire_seq`). Because the session is terminal immediately after,
|
|
nothing will ever look up that ID; `record_kernel_tx_timestamp`'s
|
|
"unknown ID -> ignore" rule discards it safely.
|
|
|
|
## MSG_ERRQUEUE drains
|
|
|
|
`receive::linux::error_queue` decodes one `MSG_ERRQUEUE` record at a time
|
|
(`try_recv_error_queue_record`) and classifies it: a genuine send-timestamp
|
|
completion (`TxTimestamp`), a timestamp-origin record that is not a usable
|
|
`SND` completion (`MalformedOrUnsupportedTimestamp` — never treated as
|
|
fatal, since this client only ever requests `TX_SOFTWARE`/`SND`, so anything
|
|
else on that origin is unexpected metadata, not a network error), a genuine
|
|
non-timestamp socket/network error (`SocketError`), or nothing (`Ignored`).
|
|
Origin (`SO_EE_ORIGIN_TIMESTAMPING` or not) is the only thing that decides
|
|
between the last two categories.
|
|
|
|
`receive::linux::drain_tx_timestamps` reads up to
|
|
`MAX_TX_TIMESTAMP_RECORDS_PER_DRAIN` (32) records per call: nonblocking, one
|
|
record at a time, no heap allocation, stopping early once the queue is
|
|
empty. It is a starvation guard, not a queue-capacity promise — one TX
|
|
timestamp is generated per successful probe, and drains happen at several
|
|
natural choke points, so a modest constant comfortably covers realistic
|
|
backlog. `Client`/`AsyncClient` call it after a successful probe send,
|
|
before processing an inbound reply, and before evaluating timeouts; it never
|
|
waits for a timestamp; an absent timestamp after all of those opportunities
|
|
simply stays absent. A genuine `SocketError` record stops the drain and is
|
|
surfaced through the ordinary `ClientError::Socket` path.
|
|
|
|
The Tokio adapter uses the same raw nonblocking `recvmsg(MSG_ERRQUEUE |
|
|
MSG_DONTWAIT)` rather than `try_io`/`poll_recv_ready`: error-queue readiness
|
|
is not ordinary read readiness, and driving it through the Tokio reactor
|
|
risks clearing readiness state that the normal Echo receive path depends on.
|
|
There is no background task or second `AsyncFd` around the Tokio-owned
|
|
socket.
|
|
|
|
## Retained state
|
|
|
|
`PendingProbe` carries an optional `kernel_tx_timestamp: Option<SystemTime>`
|
|
alongside the authoritative userspace `sent_at`. It survives pending ->
|
|
timed-out retention (`SessionMachine::record_kernel_tx_timestamp` updates
|
|
whichever of `PendingMap`/`TimedOutMap` still holds the probe) and is
|
|
otherwise bounded by the same limits those maps already enforce — no new
|
|
unbounded state. First valid observation wins on duplicates. **It is
|
|
currently dormant metadata only**: no RTT, OWD, or IPDV computation reads
|
|
it. `sent_at` remains the sole input to all of those, exactly as before this
|
|
capability existed. A later change may select it as a preferred send
|
|
timestamp; until then, treat any code path that reads
|
|
`kernel_tx_timestamp` for a measurement as a bug.
|