irtt-rs/crates/irtt-client/AGENTS.md

# `irtt-client` guidance

Repository-wide guidance in the root `AGENTS.md` also applies here.

## Probe-send commitment API

`Client::send_probe` and `AsyncClient::send_probe` return
`Result<SendReceipt, SendProbeError>`; the Tokio poll driver returns the same
result inside `Poll`. A receipt represents an Echo datagram accepted by the
socket and already committed in session state: sequence and sent count advanced,
and the probe entered pending tracking. It contains `seq`, `remote`, the public
post-send `sent_at`, accepted `bytes`, and `send_call`, without schedule metadata.
`ClientEvent::from(receipt)` constructs `EchoSent` with no schedule annotations.

Classify every failure explicitly: `NotCommitted(ClientError)` means no datagram
was accepted and no probe committed; `AfterCommit { receipt, source }` preserves
the authoritative accepted send when length validation or TX error-queue
processing fails. There is no blanket `From<ClientError>` conversion. Do not
roll back a committed probe or classify a post-send error with an indiscriminate
`?`.

All fallible state/capacity/counter/allocation and timeout preparation precedes
submission. Immediately after `send`/`try_send` returns `Ok(bytes)`, capture the
public post-send timestamp, commit infallibly, and construct the receipt before
fallible post-send processing. No suspension, readiness wait, fallible allocation,
or fallible state operation may intervene between acceptance and commitment;
the Tokio poll must never return `Pending` after acceptance.

The managed driver consumes receipts for cadence/stagger acceptance and constructs
`EchoSent` with its scheduled timestamp and timer error. A post-commit failure
advances cadence and publishes that send before failure cleanup; the underlying
cause remains a sending failure. `NotCommitted` and `Pending` advance neither
cadence nor stagger gating. Counters are cumulative accounting, never evidence
of one operation's commitment.

Tokio retains one prepared probe across cancellation/readiness false positives
and `WouldBlock`, samples a fresh private anchor on each retry, and clears the
prepared probe after commitment even when post-send processing fails. Failed
submissions still invalidate kernel TX ID correlation as documented below.

## Linux kernel TX timestamp capture

On Linux, with the `ancillary` feature, the client best-effort upgrades its
socket from RX-only `SO_TIMESTAMPING` to RX+TX immediately after a successful
Open, before the first Echo probe can be sent:

```
socket created
    -> RX timestamping configured (RX_SOFTWARE | SOFTWARE)
    -> Open attempts/retries (untimestamped)
    -> successful Open
    -> best-effort enable TX timestamping (+ TX_SOFTWARE | OPT_ID | OPT_TSONLY)
    -> first Echo probe
```

`TX_SOFTWARE` asks the kernel for a software send timestamp; `OPT_ID` asks it
to tag each notification with an automatically incrementing per-socket ID;
`OPT_TSONLY` means the notification never needs to carry the original
payload, so a zero-length receive buffer is enough. The upgrade is entirely
best-effort: if it fails, the client keeps operating on the original RX-only
configuration (`try_enable_tx_timestamping` restores it explicitly) and
timing falls back to userspace `sent_at` as it always has. TX capability
failure is never a public error and is never logged from the library.

### Best-effort kernel-ID correlation

The client uses the kernel's automatic `OPT_ID` as the probe's wire-sequence
correlation ID — no separate correlation map. On the normal successful-send
path, both counters start at zero after Open and advance together. This is
not a permanent identity: a kernel ID can theoretically be consumed while a
send is being prepared and then survive a subsequent send failure, leaving
kernel IDs one or more steps ahead of `next_wire_seq`. Do not restore an
assumption that failed or nonblocking sends cannot cause that desynchrony.

The effect is confined to the optional timing enhancement. A kernel timestamp
whose ID cannot be matched to a pending or retained timed-out probe is
discarded, as is any implausible timestamp; `sent_at` remains the ordinary
userspace fallback. **After every failed probe submission, disable kernel-ID
correlation for the rest of that session.** A gap means an otherwise plausible
later ID can name the wrong probe, so subsequent error-queue timestamps must
be discarded rather than attached by ID. This is safe because:

- **Normal successful sends track together.** `wire_seq` starts at 0 on every
  successful Open (`SessionMachine::commit_open` builds a fresh
  `ActiveSession` with `next_wire_seq: 0`) and advances by exactly 1 only via
  `commit_probe_sent`, after a confirmed successful socket send. TX
  timestamping is enabled after that same successful Open and before any probe
  is sent, so the normal zero points line up.
- **One socket, one session, no reopen.** `SessionMachine`'s state machine
  only reaches `Open` from `Connected`, and `commit_open`/`commit_local_close`
  never route back to `Connected`. A `Client`/`AsyncClient` cannot reopen a
  second session on the same socket, so there is no second epoch of wire
  sequence numbers to collide with a stale kernel ID.
- **The one extra send (Close) is harmless.** `Close` sends one datagram
  after TX timestamping is already enabled, generating one TX timestamp
  record with no corresponding `wire_seq` (`Close` does not advance
  `next_wire_seq`). Because the session is terminal immediately after,
  nothing will ever look up that ID; `record_kernel_tx_timestamp`'s
  "unknown ID -> ignore" rule discards it safely.

## MSG_ERRQUEUE drains

`receive::linux::error_queue` decodes one `MSG_ERRQUEUE` record at a time
(`try_recv_error_queue_record`) and classifies it: a genuine send-timestamp
completion (`TxTimestamp`), a timestamp-origin record that is not a usable
`SND` completion (`MalformedOrUnsupportedTimestamp` — never treated as
fatal, since this client only ever requests `TX_SOFTWARE`/`SND`, so anything
else on that origin is unexpected metadata, not a network error), a genuine
non-timestamp socket/network error (`SocketError`), or nothing (`Ignored`).
Origin (`SO_EE_ORIGIN_TIMESTAMPING` or not) is the only thing that decides
between the last two categories.

`receive::linux::drain_tx_timestamps` reads up to
`MAX_TX_TIMESTAMP_RECORDS_PER_DRAIN` (32) records per call: nonblocking, one
record at a time, no heap allocation, stopping early once the queue is
empty. It is a starvation guard, not a queue-capacity promise — one TX
timestamp is generated per successful probe, and drains happen at several
natural choke points, so a modest constant comfortably covers realistic
backlog. `Client`/`AsyncClient` call it after a successful probe send,
before processing an inbound reply, and before evaluating timeouts; it never
waits for a timestamp; an absent timestamp after all of those opportunities
simply stays absent. A genuine `SocketError` record stops the drain and is
surfaced as `ClientError::Socket`. During post-send processing it is wrapped in
`SendProbeError::AfterCommit` with the accepted send's receipt; receive/timeout
operations retain the ordinary `ClientError` path.

The Tokio adapter uses the same raw nonblocking `recvmsg(MSG_ERRQUEUE |
MSG_DONTWAIT)` rather than `try_io`/`poll_recv_ready`: error-queue readiness
is not ordinary read readiness, and driving it through the Tokio reactor
risks clearing readiness state that the normal Echo receive path depends on.
There is no background task or second `AsyncFd` around the Tokio-owned
socket.

### Testing kernel timestamp behavior

The requirements above describe production behavior, not a required unit-test
architecture.

Prefer Linux tests that exercise the real socket and kernel timestamp path
through production APIs. A test intended to prove TX timestamp support must
actually require and observe a kernel TX timestamp when running on a supported
Linux environment; accepting the userspace fallback does not prove that path.

Likewise, prefer observable socket/client behavior for genuine error-queue
errors and bounded draining where it can reasonably be induced.

Do not restore synthetic `ControlMessageOwned` permutations, classifier-unit
suites, raw ancillary fixtures, test-only socket hooks, or other internal seams
solely to cover every `MSG_ERRQUEUE` branch. `nix` owns ancillary-message
decoding and ABI safety; `irtt-client` tests should primarily prove its own
observable interpretation and behavior through the real path.

If some invariant cannot reasonably be induced through a production interface,
a small focused internal test may be retained, but the burden is on that test to
justify why integration coverage is insufficient.

## Retained state

`PendingProbe` carries an optional `kernel_tx_timestamp: Option<SystemTime>`
alongside the userspace `sent_at`. It survives pending -> timed-out retention
(`SessionMachine::record_kernel_tx_timestamp` updates whichever of
`PendingMap`/`TimedOutMap` still holds the probe) and is otherwise bounded by
the same limits those maps already enforce — no new unbounded state. First
valid observation wins on duplicates.

### Pre-send anchor vs. post-send `sent_at` (F-04)

The probe send transaction splits into two timestamps around the socket
send, not one:

- A **private pre-send anchor**, sampled immediately before the fallible
  commit preflight (`SessionMachine::finalize_probe_commit`) that must
  happen before the socket send: timeout deadline `checked_add` arithmetic
  (so `DurationOverflow` still guarantees nothing was transmitted) and the
  kernel TX lower-plausibility bound. Its monotonic half becomes
  `PendingProbe::timeout_at`'s basis; its wall half becomes
  `PendingProbe::tx_not_before_wall`. It is never exposed publicly.
- The **public `sent_at`**, a paired wall/monotonic `ClientTimestamp`
  captured immediately after the socket send returns successfully
  (`SessionMachine::commit_probe_sent`, which is infallible — all fallible
  work already happened against the pre-send anchor). This is the RTT
  endpoint, the userspace upstream-OWD fallback endpoint, `timer_error`'s
  basis, and the value surfaced on `EchoSent`/`EchoReply`/`EchoLoss`/
  `LateReply`.

So `timeout_at` is deliberately **not** derived from `sent_at.mono` — it
remains anchored to before the send, on purpose, so the existing pre-send
timeout/overflow/no-send contract is unchanged. Only the *measurement*
`sent_at` moved to after the send. A `WouldBlock` `try_send` (Tokio) never
samples a `sent_at`, never calls `finalize_probe_commit`'s matching
`commit_probe_sent`, and never advances the machine; the managed runner advances cadence only
  after an accepted send; each retry
gets its own fresh pre-send anchor, and only a successful attempt's anchor
and `sent_at` are ever retained.

**The kernel TX timestamp is eligible for exactly one measurement: the
upstream (`client_to_server`) one-way delay send endpoint.**
`compute_one_way`'s private `preferred_send_wall` helper prefers it over the
userspace `sent_at.wall` sample when

    tx_not_before_wall <= kernel_tx_timestamp <= reply_received_at.wall

(inclusive, compared only against the client's own wall clock — never
against the remote server's wall timestamp). The lower bound is the
pre-send `tx_not_before_wall`, not the post-send `sent_at.wall`: a
legitimate `TX_SOFTWARE` timestamp can be generated while the send syscall
is still in flight, so it can legitimately precede the post-send `sent_at`
sample without being implausible. Otherwise, including when no kernel
timestamp was observed, the upstream endpoint falls back to `sent_at.wall`
(the post-send sample); a rejected observation is never erased from
`PendingProbe`, only skipped for that measurement. There is no
maximum-lag bound: unlike the kernel receive side, legitimate client
send-to-reply time can exceed a second.

`sent_at` remains the RTT endpoint and the userspace fallback for the
upstream (`client_to_server`) one-way delay, exactly as before this
capability existed. It is not an input to the downstream
(`server_to_client`) one-way delay, which is derived solely from
`ReceiveMeta::preferred_receive_wall` and the negotiated server send
timestamp — unaffected by this capability or by F-04. `TX_SOFTWARE` is
still a software kernel timestamp near the driver handoff, not a NIC
hardware transmit time. `irtt-stats`'s `ReplySample`
copies public `sent_at` straight into `client_send_mono`/`client_send_wall_ns`,
which `send_ipdv_ns` compares across probes for the upstream IPDV component —
so that component shifts endpoint along with `sent_at` itself (pre-send vs.
post-send) exactly as RTT and the upstream OWD fallback do; it is not a
separate invariant. A normal
reply and a measurable late reply (one whose `PendingProbe` is still
retained in `TimedOutMap`) apply the identical selection policy; an
untracked late reply has no retained probe and stays unmeasurable, and a
kernel timestamp that arrives after a reply has already been processed is
simply never observed by that reply.