irtt-rs/docs/INTEROP_COMPARISON.md

Interop comparison

Feature comparison

This compares irtt-rs with the original heistp/irtt release v0.9.2 (2026-07-17). The aim is for the two implementations to interoperate, though the interop tests and verified protocol baseline still use upstream v0.9.1. Both upstream releases use wire protocol version 1, but v0.9.2 hasn't been run through this project's interop harness. See docs/protocol/ for the verified protocol baseline.

Capability	irtt-rs	Upstream IRTT (v0.9.2)
Wire protocol	Version 1, verified against upstream v0.9.1	Version 1, unchanged from v0.9.1
Finite probing	Default duration of 10s	Default duration of 1m
Continuous probing	--continuous or --duration 0, with optional per-target reconnection	Streaming mode (-s, added in v0.9.2)
Statistics in continuous mode	Optional count/min/max/mean/variance per session, with bounded IPDV tracking and no exact medians. Disable with --no-stats or omit the stats feature	No statistics; raw/event output only to avoid unbounded memory
Multiple targets	Labels and staggered/burst pacing in one invocation	One remote address per invocation
Terminal UI	Live graph and dashboard through irtt-tui	None
Output formats	table, csv, tsv, jsonl, with selectable columns	Text summary, JSON (optionally gzip to file), and fixed-field raw output; no CSV/TSV or column selection
Replay saved results	Not implemented	report subcommand for saved JSON results
Benchmarking and clock diagnostics	Not implemented	Benchmarking and timer/clock-diagnostic subcommands
Client payload and timing controls	No client-side payload fill or don't-fragment/wait-strategy/timer-algorithm controls	Client-side payload fill and controls for don't-fragment, wait strategy, and timer algorithm
Kernel timestamps	Best-effort preference on Linux for its own send/receive endpoint	Not implemented
Server listeners	Several listeners in one process, supervised by ServerSet	Multiple requested addresses through ListenAndServe
Server wildcard reply-source selection	Default on Linux/macOS/FreeBSD; other platforms refuse wildcard binds	Opt-in through --set-src-ip, with a documented per-packet allocation cost
Server default bind	Wildcard addresses for both families, subject to the platform restriction above	Wildcard addresses for both families; no documented platform restriction
Server payload fill policy	Honors the session's negotiated request, with a fixed fallback pattern; no options to restrict or override it	Options to restrict or override client fill requests
Syslog logging	Not implemented	Supported
Bundled service files	None	systemd and OpenRC
Client/server library API	Separate, independently versioned Rust crates (irtt-client, irtt-server)	One Go package exposing both roles
Prebuilt release binaries	Published for six target triples	None published; install through go install or build manually

Black-box testing

scripts/compare_irtt_clients.py is a black-box comparison harness for running upstream irtt and irtt-rs clients concurrently against the same server. It captures raw stdout/stderr, upstream JSON, parsed RTT samples, per-case summaries, and comparison JSON under target/interop-comparisons by default.

The harness does not inspect upstream source or tests. It only invokes the upstream executable.

Remote or Existing Local Server

Build irtt-rs and ensure an upstream irtt executable is available:

cargo build -p irtt-rs --bin irtt-client
go install github.com/heistp/irtt/cmd/irtt@v0.9.1

Run the default comparison set:

python3 scripts/compare_irtt_clients.py --upstream-irtt "$HOME/go/bin/irtt"

To compare against a specific target:

python3 scripts/compare_irtt_clients.py \
  --target netperf-eu.bufferbloat.net \
  --duration 15s \
  --interval 250ms \
  --upstream-irtt "$HOME/go/bin/irtt"

Remote comparisons are useful for smoke coverage, but Internet paths are noisy and can make RTT/loss differences hard to interpret.

Linux Netem Interop

scripts/run_netem_interop_ci.py provides deterministic Linux CI coverage using network namespaces, a veth pair, and tc netem. It is intended for GitHub Actions Ubuntu runners or a Linux host with passwordless sudo for ip and tc.

The harness creates two namespaces:

• irtt-client with veth-client at 10.10.0.1/24
• irtt-server with veth-server at 10.10.0.2/24

It runs the upstream server in the server namespace:

irtt server -b 10.10.0.2:2112 --tstamp=dual

Both clients run in the client namespace through the existing comparison harness:

sudo ip netns exec irtt-client ...

Netem qdiscs are applied on veth egress:

• veth-client controls client-to-server delay/loss
• veth-server controls server-to-client delay/loss

Cleanup deletes qdiscs, stops the server process, and removes both namespaces in a finally path.

Scenarios

The CI harness implements:

scenario	client-to-server	server-to-client	duration	interval	policy
baseline-no-netem	none	none	5s	100ms	deterministic
symmetric-20ms	delay 20ms	delay 20ms	5s	100ms	deterministic
symmetric-50ms-jitter	delay 50ms 5ms	delay 50ms 5ms	5s	100ms	jitter
asymmetric-delay	delay 40ms	delay 10ms	5s	100ms	deterministic
packet-loss	loss 5%	loss 5%	10s	50ms	stochastic loss

Asymmetric one-way delay fields include client/server wall-clock offset. Since both clients run in the same client namespace against the same server namespace, relative one-way shape is still useful for diagnostics, but the CI pass/fail policy is based on RTT and packet behavior.

Pass/Fail Policy

Deterministic scenarios fail when:

• either client exits non-zero
• parsed mean RTT or received packet counts are missing
• either client receives no packets
• unexpected loss exceeds 0.5%
• mean RTT delta exceeds max(2000us, 5% of expected RTT)

The jitter scenario fails on crashes, parser failures, no received packets, or mean RTT delta above max(8000us, 10% of expected RTT).

The packet-loss scenario fails on crashes, parser failures, or no received packets. Loss-rate deltas above 10% are warnings because packet loss is stochastic and exact percentages are not stable enough for CI.

Thresholds can be adjusted with:

python3 scripts/run_netem_interop_ci.py \
  --mean-abs-tolerance-us 3000 \
  --mean-pct-tolerance 0.08 \
  --jitter-mean-abs-tolerance-us 10000 \
  --loss-rate-warning-pct 12

Manual Linux Usage

On Linux:

cargo build -p irtt-rs --bin irtt-client
go install github.com/heistp/irtt/cmd/irtt@v0.9.1
python3 scripts/run_netem_interop_ci.py \
  --all \
  --skip-build \
  --upstream-irtt "$HOME/go/bin/irtt" \
  --irtt-rs-command "$PWD/target/debug/irtt-client"

Run one scenario:

python3 scripts/run_netem_interop_ci.py --scenario symmetric-20ms

Artifacts are written under target/interop-netem/<timestamp>/ by default. Each scenario contains the existing comparison harness output under <scenario>/comparison/<timestamp>/, plus netem-specific summary.md and netem-summary.json files. Server stdout/stderr are captured at the run root as server.stdout and server.stderr.

On macOS, ip netns and tc netem are unavailable; use the Gitea Actions workflow for netem coverage.