Nils Andreas Svee
3a29b4efb2
Rust CI / cargo fmt (push) Successful in 45s
Rust CI / cargo check (x86_64-unknown-linux-gnu) (push) Successful in 1m4s
Rust CI / cargo clippy (push) Successful in 1m6s
Rust CI / cargo test (x86_64-unknown-linux-gnu) (push) Successful in 1m48s
Rust CI / cargo check (x86_64-unknown-linux-musl) (push) Successful in 50s
Rust CI / cargo doc (push) Failing after 48s
Rust CI / cargo test (x86_64-unknown-linux-musl) (push) Successful in 1m49s
Proto Fuzz Smoke / cargo fuzz smoke (push) Successful in 2m3s
Supply chain / cargo-deny advisories (push) Failing after 29s
Netem Interop / netem interop (push) Successful in 1m33s
Supply chain / cargo-deny bans licenses sources (push) Failing after 27s
Supply chain / actionlint (push) Failing after 14s
144 lines
4.4 KiB
Markdown
144 lines
4.4 KiB
Markdown
# Interop Comparison
|
|
|
|
`scripts/compare_irtt_clients.py` is a black-box comparison harness for running
|
|
upstream `irtt` and `irtt-rs` clients concurrently against the same server. It
|
|
captures raw stdout/stderr, upstream JSON, parsed RTT samples, per-case
|
|
summaries, and comparison JSON under `target/interop-comparisons` by default.
|
|
|
|
The harness does not inspect upstream source or tests. It only invokes the
|
|
upstream executable.
|
|
|
|
## Remote or Existing Local Server
|
|
|
|
Build `irtt-rs` and ensure an upstream `irtt` executable is available:
|
|
|
|
```sh
|
|
cargo build -p irtt-rs --bin irtt-client
|
|
go install github.com/heistp/irtt/cmd/irtt@v0.9.1
|
|
```
|
|
|
|
Run the default comparison set:
|
|
|
|
```sh
|
|
python3 scripts/compare_irtt_clients.py --upstream-irtt "$HOME/go/bin/irtt"
|
|
```
|
|
|
|
To compare against a specific target:
|
|
|
|
```sh
|
|
python3 scripts/compare_irtt_clients.py \
|
|
--target netperf-eu.bufferbloat.net \
|
|
--duration 15s \
|
|
--interval 250ms \
|
|
--upstream-irtt "$HOME/go/bin/irtt"
|
|
```
|
|
|
|
Remote comparisons are useful for smoke coverage, but Internet paths are noisy
|
|
and can make RTT/loss differences hard to interpret.
|
|
|
|
## Linux Netem Interop
|
|
|
|
`scripts/run_netem_interop_ci.py` provides deterministic Linux CI coverage using
|
|
network namespaces, a veth pair, and `tc netem`. It is intended for GitHub
|
|
Actions Ubuntu runners or a Linux host with passwordless `sudo` for `ip` and
|
|
`tc`.
|
|
|
|
The harness creates two namespaces:
|
|
|
|
- `irtt-client` with `veth-client` at `10.10.0.1/24`
|
|
- `irtt-server` with `veth-server` at `10.10.0.2/24`
|
|
|
|
It runs the upstream server in the server namespace:
|
|
|
|
```sh
|
|
irtt server -b 10.10.0.2:2112 --tstamp=dual
|
|
```
|
|
|
|
Both clients run in the client namespace through the existing comparison
|
|
harness:
|
|
|
|
```sh
|
|
sudo ip netns exec irtt-client ...
|
|
```
|
|
|
|
Netem qdiscs are applied on veth egress:
|
|
|
|
- `veth-client` controls client-to-server delay/loss
|
|
- `veth-server` controls server-to-client delay/loss
|
|
|
|
Cleanup deletes qdiscs, stops the server process, and removes both namespaces in
|
|
a `finally` path.
|
|
|
|
### Scenarios
|
|
|
|
The CI harness implements:
|
|
|
|
| scenario | client-to-server | server-to-client | duration | interval | policy |
|
|
| --- | --- | --- | --- | --- | --- |
|
|
| `baseline-no-netem` | none | none | `5s` | `100ms` | deterministic |
|
|
| `symmetric-20ms` | `delay 20ms` | `delay 20ms` | `5s` | `100ms` | deterministic |
|
|
| `symmetric-50ms-jitter` | `delay 50ms 5ms` | `delay 50ms 5ms` | `5s` | `100ms` | jitter |
|
|
| `asymmetric-delay` | `delay 40ms` | `delay 10ms` | `5s` | `100ms` | deterministic |
|
|
| `packet-loss` | `loss 5%` | `loss 5%` | `10s` | `50ms` | stochastic loss |
|
|
|
|
Asymmetric one-way delay fields include client/server wall-clock offset. Since
|
|
both clients run in the same client namespace against the same server namespace,
|
|
relative one-way shape is still useful for diagnostics, but the CI pass/fail
|
|
policy is based on RTT and packet behavior.
|
|
|
|
### Pass/Fail Policy
|
|
|
|
Deterministic scenarios fail when:
|
|
|
|
- either client exits non-zero
|
|
- parsed mean RTT or received packet counts are missing
|
|
- either client receives no packets
|
|
- unexpected loss exceeds `0.5%`
|
|
- mean RTT delta exceeds `max(2000us, 5% of expected RTT)`
|
|
|
|
The jitter scenario fails on crashes, parser failures, no received packets, or
|
|
mean RTT delta above `max(8000us, 10% of expected RTT)`.
|
|
|
|
The packet-loss scenario fails on crashes, parser failures, or no received
|
|
packets. Loss-rate deltas above `10%` are warnings because packet loss is
|
|
stochastic and exact percentages are not stable enough for CI.
|
|
|
|
Thresholds can be adjusted with:
|
|
|
|
```sh
|
|
python3 scripts/run_netem_interop_ci.py \
|
|
--mean-abs-tolerance-us 3000 \
|
|
--mean-pct-tolerance 0.08 \
|
|
--jitter-mean-abs-tolerance-us 10000 \
|
|
--loss-rate-warning-pct 12
|
|
```
|
|
|
|
### Manual Linux Usage
|
|
|
|
On Linux:
|
|
|
|
```sh
|
|
cargo build -p irtt-rs --bin irtt-client
|
|
go install github.com/heistp/irtt/cmd/irtt@v0.9.1
|
|
python3 scripts/run_netem_interop_ci.py \
|
|
--all \
|
|
--skip-build \
|
|
--upstream-irtt "$HOME/go/bin/irtt" \
|
|
--irtt-rs-command "$PWD/target/debug/irtt-client"
|
|
```
|
|
|
|
Run one scenario:
|
|
|
|
```sh
|
|
python3 scripts/run_netem_interop_ci.py --scenario symmetric-20ms
|
|
```
|
|
|
|
Artifacts are written under `target/interop-netem/<timestamp>/` by default. Each
|
|
scenario contains the existing comparison harness output under
|
|
`<scenario>/comparison/<timestamp>/`, plus netem-specific `summary.md` and
|
|
`netem-summary.json` files. Server stdout/stderr are captured at the run root as
|
|
`server.stdout` and `server.stderr`.
|
|
|
|
On macOS, `ip netns` and `tc netem` are unavailable; use the Gitea Actions
|
|
workflow for netem coverage.
|