From 208341a84dc759473332ce7ce701cd2725f7bd85 Mon Sep 17 00:00:00 2001 From: Nils Andreas Svee Date: Tue, 28 Apr 2026 15:36:40 +0700 Subject: [PATCH] first revision plan + docs --- .gitignore | 11 + captures/basic-session.pcapng | Bin 0 -> 1444 bytes captures/dscp-46.pcapng | Bin 0 -> 968 bytes captures/dscp-session.pcapng | Bin 0 -> 1140 bytes captures/dscp-values.pcapng | Bin 0 -> 1408 bytes captures/full-session.pcapng | Bin 0 -> 1380 bytes captures/hmac-session.pcapng | Bin 0 -> 1244 bytes captures/large-packet.pcapng | Bin 0 -> 3716 bytes captures/long-sfill.pcapng | Bin 0 -> 3848 bytes captures/no-test.pcapng | Bin 0 -> 568 bytes captures/tstamp-none.pcapng | Bin 0 -> 2160 bytes docs/BLACKBOX_VERIFICATION_REPORT.md | 610 ++++++++ docs/CLEANROOM_NOTES.md | 189 +++ docs/IMPLEMENTATION_PLAN.md | 2036 ++++++++++++++++++++++++++ docs/IRTT_CLIENT_PROTOCOL_SPEC.md | 1450 ++++++++++++++++++ docs/RUST_DESIGN.md | 1452 ++++++++++++++++++ docs/test-vectors.md | 304 ++++ 17 files changed, 6052 insertions(+) create mode 100644 .gitignore create mode 100644 captures/basic-session.pcapng create mode 100644 captures/dscp-46.pcapng create mode 100644 captures/dscp-session.pcapng create mode 100644 captures/dscp-values.pcapng create mode 100644 captures/full-session.pcapng create mode 100644 captures/hmac-session.pcapng create mode 100644 captures/large-packet.pcapng create mode 100644 captures/long-sfill.pcapng create mode 100644 captures/no-test.pcapng create mode 100644 captures/tstamp-none.pcapng create mode 100644 docs/BLACKBOX_VERIFICATION_REPORT.md create mode 100644 docs/CLEANROOM_NOTES.md create mode 100644 docs/IMPLEMENTATION_PLAN.md create mode 100644 docs/IRTT_CLIENT_PROTOCOL_SPEC.md create mode 100644 docs/RUST_DESIGN.md create mode 100644 docs/test-vectors.md diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..1879c3d --- /dev/null +++ b/.gitignore @@ -0,0 +1,11 @@ +# Generated by Cargo +# will have compiled files and executables +target/ +*/target/ + +# Remove Cargo.lock from gitignore if creating an executable, leave it for libraries +# More information here https://doc.rust-lang.org/cargo/guide/cargo-toml-vs-cargo-lock.html +Cargo.lock + +# These are backup files generated by rustfmt +**/*.rs.bk diff --git a/captures/basic-session.pcapng b/captures/basic-session.pcapng new file mode 100644 index 0000000000000000000000000000000000000000..ead1d084d516ffbd61fd8d94063e2a1e368209b5 GIT binary patch literal 1444 zcmb7^PiPZC6vkhZP2E)M-wLG)4uUi>H8k68Qc*N&M3j^kU8oul-Rwr(Cfi-MiMF-q zK|JInh(<4-)dY@AtkXZ#p_VJMIC% z@Znfb8(n|95daa6*>HCea8K9!u#I~F=krD|BU{j0zA_jR_Zw>{G z8>8)zyO32b(LH8>=1G9_A9ezyw*lOwZDKBm#582q$GRTpKz{x7%d@o|-PnICfy{r| z`HOyb>baLKrk6u5JMcFfdkwk`OwS!XGh^1fS0{wVC;j<~>1ds49{Y4H=kslR^Rjuq zk=sh}*h?Jt!dEYw=ey~7gWG_(`2IHfKJcizh>xDUWBTDd(>(T_l?k8kmqjm|=ezl> z|JOT@J-+H^*Kc{*Jm1I`Xl`-gTeIVD(f6Ujec$oXXp!lKvrO~YAA4qfzH7Z+HqSS5 zTL~UJw#(1{eB8_C`9^k#<{r0!^nG}7xPRuF|gM5T2*O--EhLHOqL7I6tNxas-YKjlAj?u hyA1J8#k010M8UfZ_n+=p6@l)j#dy zJG~Vqy1zgfyA)w~C3+>O;DvvHyA;H0h3a|I78XA)_P~>zlcdk@yy=^PWO87Qh~`g@ z9ggt%S5FFQbjI^)RWWZwX^iT2<>G=ca{8p685iZ2tU1EST1Lhwh?Hgfw`GHSzB`eo z7)_GZs(TgN6RDE)tILbFzmyiHp3_Z{D$1s)xb-@pZssedoi_`4)0s*$od~mFjyq@q zX{1n;YVHIdHJWo>uWVPA_V80G-``97h+2*(JlAguBa@jT8r$ELkadj>A?wF}Lr!xI z_p)-Eo8heOEN30-WH~DmQIyr^pSKdx;22j`HKA%8h)iTVYzWz}4jA$**KjX;=Emi4 zwpQHE(rD)tXGcpj$@1*_o7ewiARCMoHU!4XkRic_1BOmls^7zmV1BDpVg~gY`2B7K z=AZ13|6srT5M~GK>#~)dZ0sc%T0Qvq>%02Kp$m-K#%{yzxoesI+N zYY$kD!t7w(y*3%Ty?(pnJ Y{M#b-JLKSXR!-S(cnO>rA&T3zu7YJ*(xjg?!eSN+BJ9&O+z7 zM}-#>2oQyuJ&vCelx*9nYWn;h+$G}my)XpOFdXLCZk;KaOb%e*F3(FC;*sAR6&MdL zi(sCw@DrGm1k^cz{F7~fGy4G^;xS3>4M~Ngo!s!tn~{hhEN{O4J``Pkwe>bWAc|sK zl)ed`CwUtv3i8&bE4+v~Deyl1>r2>MQrjDl?wmsJXriD_9#|Uv@h=8>6JrTQL5w4% z3a5TZ9^(yM|Afu-oL}vfkpW!>-M^b4=1%rXAGY{2%=Y^8*}~s!nm>u6@M8VzJF9hY z4QM7s70BZaez!GKup)(T~0LxR-r#GtBPSH?e6((E$AM z+T5}C*!TL(^BeTlTFbBRuljYDT?+QoykXVkV_1`d-&As{9A9q%TvTxD7l@uS56CnT$$>NXo|!xU-#vH!jBjYjR|kNF z>9M0O9RFD11|BY|YBA3iq8=CnSMV4`Rn9Rk+WPCCim!^V92cW&s zJ|0tH1P^ZTKv4- zo0_C8VN#Ixt$9I5DWJ5pXFiBdZ9d&_ZdN_ zekctF56fR?oD5_CsrC{wz-G|<{RA=hvrjK}vwz^C&a*T6GuiG=Hhq6eBUU$WhfCWd zUlO0DseZj~*5xZ^ZL{ai)kfcemApQVHKld7!sn+>w$b-+`waH&wb`UWeb9MP&>6Pw z%UvE=eBspE+xm|OW_x?!*NWWb0W|c!Guqp-J@CC*)9ddxGCe(jj9wmi`#^SjfWfl# z(aA9OZ+XCGQ2+ge{T*neptGA2yV=o~PBx#${wxo4vO7J{yf&AMpa)L)>(}Z2h-lWa zQ)X?mJ3Zi6w!1vAd)~=5`nEh^vq@tBvxrGSeZ2a;qr2cZ%qi7UC9jp4sws~OQeI-& j4JMgCI{IJIS}kx=r~eeq1Nfd7=RQ*kb$M%M4#fBisrU-B literal 0 HcmV?d00001 diff --git a/captures/full-session.pcapng b/captures/full-session.pcapng new file mode 100644 index 0000000000000000000000000000000000000000..d6aa5d4324fddcb98957ada5a1eeca3513febdd3 GIT binary patch literal 1380 zcmb7^&ui0Q7{{NcU7F2qAdVqJ@jbbgjk2cM+TjirD}rrN6v>7?G)+3%^+(dI)lEdJ zpdKA#DtOsNP&Wke;IEVSUG(D7^Yjl8nOJ?ECIw@2un#L6B#(uB&vd`9?Y`3|)GyClZ-i}8DcX947 z$7deUtyad;uVhPi*=c(}opWw&U7kWsX`l1pnA B{qFz( literal 0 HcmV?d00001 diff --git a/captures/hmac-session.pcapng b/captures/hmac-session.pcapng new file mode 100644 index 0000000000000000000000000000000000000000..54e0bd1d3526d09ad0a96801505f0e67be547c54 GIT binary patch literal 1244 zcmd<$<>i{fz`)>ZqbV?jYqs)DZ(GlMomZep^3u!519p0S>xjzUst zW=@KNk*SN3p^=4xhD%~md1fAv50o{~WME-1W^gIZEl5r*P|yg^EJ`iTNG!_MR4~yq z(=$=faL+7JNY2mA1yX6|$tH>BCKl!l9W4hAL$W`>-610c!4 z;FF(Ukd&C5%?VV+29)P!;A3DYO({?)$S*2UFfueWVPN1P(Y+vRK=y&m%K&kJpe}%o zVa@Wbu^vD+2r~h(D+7beR&@pj2SEmgdZ05w_~dgACkBIm&LYdBIsZ=(TfQo|LTujQ zMOQ=a7??3KF*RIR^O=viVa2tb?5u2TZ0u|x^Fj6$05J%F?CD>zH8udq24Sc@KE;zz z?cs1b`JBPxpEuAR?hKAPk11bE_RUz|`RDkQ4!Ouv_OY*DwmW094`dI>z8)Y30g!#+ zE4Rkx0NEf6wJ#&j6V*Pjze4_{1MTCu+3w%8Sy;d@a;;8(_O18-VfKN{BL#rmfbNz> zSllwb6V)wXzeC)@_1fS-ZF1>i?#k)1s~*c9yoT%+P`UwO5cuIXk9S_~Dhc5?5pp1Y zLllG`&lCyfV{=R1ha(u_vi1wATad$La@)L(^QPIKeaZ7|<^GK`>kPpJp^kVVz0;*fU;iB@-9GG5szMkb@d2eoQTJOZA zMjm`l4`AT}GLsdEbAT8GKxRjlUx{U8kYRAnFU>1SEh<(hD9SI(Oi4{qNUBswffOR3 ivIwNscjbauP#FMXn*#N*0NLyiqk!geKv@t9WCsA6xgCK3 literal 0 HcmV?d00001 diff --git a/captures/large-packet.pcapng b/captures/large-packet.pcapng new file mode 100644 index 0000000000000000000000000000000000000000..02626b61060947f4ed3ae4e9eb55be18fe5a0eb0 GIT binary patch literal 3716 zcmeHKO=}ZT6up_Rj@1f1rz}rUAbKIgcTXguj9I69QFVx3F zD!O5U0xF1b3a=(q!Z513Ui}EZW#IFpa2%k;BNl~m!mu41;Mf8CB^lBqzBMW^o-7-1 z@(dSqm=hGlWq{&m4dCK1z*B6Ku)Sk0!l_PSWb<|=qbTjS-@go~-K*9f)eXZKFuI(? z%@GvD{Smq6F(>r6FI;=o&uuPl??AqD9=X+tXN?z|tGCAX(;#lrNH3|zu*Jns9e|K( z=v(X0`!%S4{j0i#8nCNDG&1A_Ovag#-Hc;DwW j&Z>_XPnSK~8R^~10vEKaQGNw~r(sTf6guX6gEamCHq9G@ literal 0 HcmV?d00001 diff --git a/captures/long-sfill.pcapng b/captures/long-sfill.pcapng new file mode 100644 index 0000000000000000000000000000000000000000..9afbefe931e6c4f98f6e0d4c440110f81450027c GIT binary patch literal 3848 zcmeHKPiWI%6n{;cESpX!6ddB=^JL3LY17!wp|G)vf7q%pal!;mlT<^~CSB2JtBo@9%>H05#oVj_p<%mXiV4|A+Mw4e=4) z3?*pyE{Kp{dghH`OwOU70*HMs0Zc{!ma$Co-Rn{wqOGIRmkZ%X-ycuMUObr~48rQ3 zfv`484ev5uBloc`HlO!nt*;=~WNQwwLLrW;zxw!gfN!ti`awYy#Q{_#OnmgFpxVy$+E0lm7S4P`gXpzsyjPX21To=N^BC>gqpX;K%?LIvLF}>7B1$ z2oA%vU8~>#w8$LSzG>=)&I)&!-WfP)Zl!0JcdmLgFH$Vw=fW7b-%W@mVBBlO@dK?q BQfdGI literal 0 HcmV?d00001 diff --git a/captures/no-test.pcapng b/captures/no-test.pcapng new file mode 100644 index 0000000000000000000000000000000000000000..53e99994d9e54868e5158278132e5f5dea9fdb4e GIT binary patch literal 568 zcmb7BJx{_=6uq>xRQyOxoQ%VBqro&#%2$GkD8xmgi30;Lloqw2rA?uJ4N8nZ!Nt)@ z2Ny?O++k<-2k2iw?^R56aPTCj_uRbu&bjG}&dx@!0H9KcFNUzbPr!i$MbEQatfKM| z2S+!ycbS?|Qc7}#H3Q2wnVK%CNj1k}CEY)=I@sfBLWVG;pcFWsp?fS=wft81K=<1+ z)0B*&vDl{7V}|QESoX4prf0QWRx|T5rUTF}w8t?upOAn99&9&(UkSEc*K6uVdkUvS zJfDU+fWY*a=lVUSCX*?EnSbs}F_cGsb!rf2J0e`XRvUTLL=IU7NI$Fttjz1(_fMooJW1k^u486PJ4bCv)A literal 0 HcmV?d00001 diff --git a/captures/tstamp-none.pcapng b/captures/tstamp-none.pcapng new file mode 100644 index 0000000000000000000000000000000000000000..186ecffdc42848707fdc1f705983215d46791f94 GIT binary patch literal 2160 zcmb7`O-K}R6u_Ue>+EEi85>B63?#Vbw(YK~yP^kUEyJ`H6qM3+zpNHrcU?cy2<;L? zr=%cHAsQ6bB?75S2k{_Eb}~;AL>&VAIB5#5y*D2;|DEyBgE#ZW+4;TqpWn=`dHMNy zj{u;huHtwO?)Q}7K!CHUR9}zS;x<7AB*M|lt)jcyQR#5m#mI2HFDAOH8r&|oPqa0J z(>LS&c#LPAS;xVz?!dct0h z&+CcRlw)@QnuX>Vr2{Dh;J^fZNhfXvXig?mk#My40Dg+c^9SKDz;G-jrjqF)(d}|o z0UY|zd8rNcBVHRFV2lTNs6A2;^k7ZNVV(nUeh~ob3jro@Ov=qA!4JimGuCHUb8#(L&n7-FdEK~4rEpW|vU#@+crWj+LfX6wMDAlCL)YtV-^rNZ(!pRlaOWppEED)nx1Ii9m{ z#3OD9lY+S4{23w|<{B(F%K6F^kW2_&UA3_H4$rT0h$G2N` z{s(GVzBIS&A5^|f__tO4YuSqiY{yrYEzK^oIW3#sxj?sNbUN{(NM&o?MJ$s#cOw^B zwlufmPGuXp`}ku@=WgD?vKD6db!XTaix?KKY|~Q zbvXI7YDUM>Z_e{0C6^-wC$$c2)(26KBdjr1L<@@2yQTk1jgTiYvt zPv7h0X||W($F<14IBhTTsXE;{Y=1a|@8y+lYokf&k?oexKhbT+bCk-uY+D0tFUg1E zR_)b{pU+j8xRbu4y6&Fp&Vgd^Cx`oodeVbpDxDmO$9iI7WK4|7KkW4X0o~6$=S~;> VE>m5G*O{@l5(py~Yi-cI`3uV%oz?&V literal 0 HcmV?d00001 diff --git a/docs/BLACKBOX_VERIFICATION_REPORT.md b/docs/BLACKBOX_VERIFICATION_REPORT.md new file mode 100644 index 0000000..c7b09a0 --- /dev/null +++ b/docs/BLACKBOX_VERIFICATION_REPORT.md @@ -0,0 +1,610 @@ +# IRTT Black-Box Verification Report + +**Date:** 2026-04-28 +**irtt version:** 0.9.1 (protocol version 1, json format version 1) +**OS:** macOS Darwin 25.3.0 arm64 (Apple Silicon) +**tshark version:** TShark (Wireshark) 4.6.4 +**Loopback interface:** lo0 + +--- + +## Environment Check + +``` +$ command -v irtt +/Users/lochnair/.local/bin/irtt + +$ irtt version +irtt version: 0.9.1 +protocol version: 1 +json format version: 1 +go version: go1.26.2 on darwin/arm64 + +$ command -v tshark +/opt/homebrew/bin/tshark + +$ tshark --version +TShark (Wireshark) 4.6.4 (Git commit f7c4a74874d9). +``` + +--- + +## Open Question Resolution + +### 19.1 Open Request Field Layout — RESOLVED + +**Question:** Does the open request include a connection token field (zeroed)? + +**Test setup:** Captured a full session including open exchange. + +**Command:** +``` +irtt server -b 127.0.0.1:2112 +irtt client -d 3s -i 1s 127.0.0.1:2112 -q +``` + +**Capture:** `captures/full-session.pcapng` + +**Observed open request (frame 1):** +``` +14 a7 5b 01 01 02 02 80 f8 82 ad 16 03 80 a8 d6 +b9 07 05 06 06 06 07 06 +``` +- Bytes 0-2: Magic `14 a7 5b` +- Byte 3: Flags `01` (Open) +- Bytes 4+: Parameter payload (no connection token) + +**Observed open reply (frame 2):** +``` +14 a7 5b 03 13 52 71 87 ab b6 96 78 01 02 02 80 +f8 82 ad 16 03 80 a8 d6 b9 07 05 06 06 06 07 06 +``` +- Bytes 0-2: Magic +- Byte 3: Flags `03` (Open | Reply) +- Bytes 4-11: Connection token `13 52 71 87 ab b6 96 78` +- Bytes 12+: Parameter payload + +**Conclusion:** The open request does NOT include a connection token field. +The layout is: magic (3) + flags (1) + [HMAC (16) if applicable] + params. +This is a **normative requirement** — the connection token is absent in open +requests because it has not yet been assigned. + +--- + +### 19.2 Minimum Packet Length — RESOLVED + +**Question:** What is the minimum echo packet length for various +configurations? + +**Test setup:** Multiple sessions with different --stats and --tstamp +settings, all with -l 0 (minimum length). + +**Commands:** +``` +irtt client -d 1s -i 1s --tstamp=none --stats=none 127.0.0.1 +irtt client -d 1s -i 1s --tstamp=send --clock=wall --stats=count 127.0.0.1 +irtt client -d 1s -i 1s --tstamp=receive --clock=monotonic --stats=window 127.0.0.1 +irtt client -d 1s -i 1s --tstamp=midpoint --clock=both --stats=both 127.0.0.1 +irtt client -d 1s -i 1s --tstamp=both --clock=both --stats=both 127.0.0.1 +``` + +**Capture:** `captures/tstamp-none.pcapng` + +**Observed minimum packet lengths (echo request/reply):** + +| Stats | StampAt | Clock | Packet Length | Breakdown | +|-------|---------|-------|---------------|-----------| +| none | none | - | 16 bytes | magic(3)+flags(1)+token(8)+seq(4) | +| count | send | wall | 28 bytes | +recv_count(4)+send_wall(8) | +| window | receive | mono | 32 bytes | +recv_window(8)+recv_mono(8) | +| both | midpoint | both | 44 bytes | +recv_count(4)+recv_window(8)+mid_wall(8)+mid_mono(8) | +| both | both | both | 60 bytes | +recv_count(4)+recv_window(8)+recv_wall(8)+recv_mono(8)+send_wall(8)+send_mono(8) | + +With HMAC, add 16 bytes (observed: 76 bytes for default+HMAC). + +**Conclusion:** The minimum packet length formula is: +``` +min_length = 4 (header) + 8 (token) + 4 (seq) + + [16 if HMAC] + + [4 if recv_count (stats=count or both)] + + [8 if recv_window (stats=window or both)] + + [8 per timestamp field present] +``` + +Timestamp field count depends on StampAt and Clock: +- none: 0 fields +- send/wall or receive/wall: 1 field (8 bytes) +- send/mono or receive/mono: 1 field (8 bytes) +- send/both or receive/both: 2 fields (16 bytes) +- both/wall: 2 fields (16 bytes) +- both/mono: 2 fields (16 bytes) +- both/both: 4 fields (32 bytes) +- midpoint/wall: 1 field (8 bytes) +- midpoint/mono: 1 field (8 bytes) +- midpoint/both: 2 fields (16 bytes) + +This is a **normative requirement** — implementations must compute the +correct minimum to avoid malformed packets. + +--- + +### 19.4 Varint Encoding — RESOLVED + +**Question:** Is the varint encoding byte-compatible with protobuf-style +LEB128/zigzag? + +**Test setup:** Decoded parameters from captured open request packets and +verified against known values. + +**Observed parameter encodings:** + +| Parameter | Value | Zigzag Encoded | LEB128 Bytes | +|-----------|-------|---------------|--------------| +| ProtocolVersion=1 | 1 | 2 | `02` | +| Duration=3s | 3000000000 | 6000000000 | `80 f8 82 ad 16` | +| Duration=1s | 1000000000 | 2000000000 | `80 a8 d6 b9 07` | +| Interval=1s | 1000000000 | 2000000000 | `80 a8 d6 b9 07` | +| Length=1472 | 1472 | 2944 | `80 17` | +| ReceivedStats=Both | 3 | 6 | `06` | +| StampAt=Both | 3 | 6 | `06` | +| Clock=Both | 3 | 6 | `06` | +| DSCP=0xb8 | 184 | 368 | `f0 02` | +| ServerFill (tag) | 9 | n/a (uvarint) | `09` | +| ServerFill (length) | 24 | n/a (uvarint) | `18` | + +All decoded correctly using standard protobuf-style zigzag + LEB128. + +**Verified independently with Python decoder.** + +**Conclusion:** The varint encoding is standard protobuf-compatible. This +is a **normative requirement** for interoperability. + +--- + +### 19.5 HMAC Computation Scope — RESOLVED + +**Question:** Is the HMAC computed over the entire packet (including payload) +with the HMAC field zeroed? + +**Test setup:** Captured HMAC session, extracted packet bytes, independently +computed HMAC-MD5 with Python. + +**Commands:** +``` +irtt server -b 127.0.0.1:2114 --hmac=testkey +irtt client -d 2s -i 1s --hmac=testkey 127.0.0.1:2114 -q +``` + +**Capture:** `captures/hmac-session.pcapng` + +**Verification (Python):** +```python +import hmac, hashlib +key = b"testkey" +# Zero HMAC field (bytes 4-19), compute over entire packet +computed = hmac.new(key, packet_with_zeroed_hmac, hashlib.md5).digest() +``` + +**Results:** +| Packet Type | Captured HMAC | Computed HMAC | Match | +|-------------|--------------|--------------|-------| +| Open request | `ff9016a7aa537816...` | `ff9016a7aa537816...` | YES | +| Echo request (seq 0) | `d9874f82b3131031...` | `d9874f82b3131031...` | YES | +| Echo reply (seq 0) | `eb30c07d6375a30b...` | `eb30c07d6375a30b...` | YES | +| Close request | `f5cd0fa9de9d7d66...` | `f5cd0fa9de9d7d66...` | YES | + +Also verified the spec's test vectors (18.1 and 18.2) — both match when +using the correct 92-byte packet size. + +**Conclusion:** HMAC-MD5 is computed over the entire packet buffer with the +16-byte HMAC field zeroed. This is a **normative requirement**. + +**Additional finding:** The test vectors in Section 18 are described as +"target length 256 bytes (shown truncated)" but the HMACs were actually +computed over 92-byte packets (76-byte header + 16-byte payload). The +"256 bytes" description is incorrect in the spec. + +--- + +### 19.7 Server Received Window Validity — RESOLVED + +**Question:** Is bit 0 always set for valid windows, making 0 a sentinel for +invalid? + +**Observed window values from `captures/full-session.pcapng`:** + +| Seq | Received Count | Received Window (hex LE) | Window (binary, low bits) | +|-----|---------------|-------------------------|--------------------------| +| 0 | 1 | `01 00 00 00 00 00 00 00` | ...0001 | +| 1 | 2 | `03 00 00 00 00 00 00 00` | ...0011 | +| 2 | 3 | `07 00 00 00 00 00 00 00` | ...0111 | + +**Analysis:** Bit 0 (LSB) represents the current packet, which was +obviously received (it's being replied to). This bit is always 1 for +valid windows. A window value of 0 would indicate either an invalid window +or a bug. + +**Conclusion:** Bit 0 is always set for valid windows. A window value of 0 +can be used as an invalidity sentinel. This is a **compatibility +recommendation** — implementations SHOULD treat window=0 as invalid. + +--- + +### 19.8 HMAC Mismatch Behavior — RESOLVED + +**Question:** What happens when HMAC keys don't match? + +**Commands:** +``` +irtt server -b 127.0.0.1:2120 --hmac=correctkey +irtt client --hmac=wrongkey --timeouts=500ms 127.0.0.1:2120 +irtt client --timeouts=500ms 127.0.0.1:2120 # no HMAC +``` + +**Observed behavior:** + +| Scenario | Server Log | Client Result | +|----------|-----------|--------------| +| Wrong HMAC key | `[Drop] [BadHMAC] invalid HMAC: ...` | `[OpenTimeout] no reply from server` (exit 1) | +| No HMAC to HMAC server | `[Drop] [NoHMAC] no HMAC present` | `[OpenTimeout] no reply from server` (exit 1) | + +**Conclusion:** The server silently drops packets with missing or invalid +HMAC. The client times out. This is **observed behavior** — the protocol does +not define an explicit HMAC error response. + +--- + +### 19.9 Minimum Restricted Interval Safety Floor — PARTIALLY RESOLVED + +**Question:** What safety floor does the client enforce for server-restricted +intervals? + +**Test setup:** Server with -i 500ms (minimum interval), client requests +200ms. + +**Command:** +``` +irtt server -b 127.0.0.1:2123 -i 500ms +irtt client -d 2s -i 200ms 127.0.0.1:2123 +``` + +**Observed:** `[ServerRestriction] server increased interval from 200ms to +500ms` — client exits with code 1 in strict mode. In loose mode, client +accepts 500ms interval. + +**Limitation:** This test only confirms that the client rejects server- +increased intervals in strict mode. It does not confirm the exact safety +floor value for server-decreased intervals (which would require a server +that decreases interval below the client's request). The safety floor for +decreased intervals cannot be easily tested without a cooperative server +that sends deliberately small intervals. + +**Conclusion:** Strict mode rejects any server parameter change by default. +The specific safety floor for decreased intervals remains an **open +question** that requires further testing. + +--- + +### 19.10 Maximum Parameter Buffer Size — PARTIALLY RESOLVED + +**Question:** Is there a maximum serialized parameter payload size? + +**Test setup:** Open request with all parameters including a 24-character +ServerFill string. + +**Observed:** Largest serialized parameter payload was 52 bytes (with +ProtocolVersion, Duration, Interval, Length=1472, ReceivedStats=both, +StampAt=both, Clock=both, DSCP=0xb8, ServerFill="pattern:aabbccddeeff0011"). + +**Analysis:** Even with maximum-value parameters and the longest reasonable +ServerFill string (32 bytes max), the parameter buffer stays well under +128 bytes. A rough upper bound: +- Tag+value overhead per param: ~6 bytes for large values +- 8 params × 6 bytes = 48 bytes +- Plus ServerFill: tag(1) + length(1) + string(32) = 34 bytes +- Total: ~82 bytes maximum + +**Conclusion:** The 128-byte limit stated in the spec appears safe. +Reclassified as **compatibility recommendation** — implementations should +allocate at least 128 bytes for the parameter buffer. + +--- + +### 19.11 Field Ordering Verification — RESOLVED + +**Question:** Is the field ordering in Section 8.1.3 correct? + +**Test setup:** Multiple captures with different field combinations. + +**Verified orderings:** + +1. **No optional fields (stats=none, tstamp=none):** + Token → Seq. Packet = 16 bytes. ✓ + +2. **Count + send wall:** + Token → Seq → Recv Count → Send Wall. Packet = 28 bytes. ✓ + +3. **Window + receive mono:** + Token → Seq → Recv Window → Recv Mono. Packet = 32 bytes. ✓ + +4. **Both + midpoint both:** + Token → Seq → Recv Count → Recv Window → Midpoint Wall → Midpoint Mono. + Packet = 44 bytes. ✓ + +5. **Both + both both (default):** + Token → Seq → Recv Count → Recv Window → Recv Wall → Recv Mono → + Send Wall → Send Mono. Packet = 60 bytes. ✓ + +6. **HMAC + both + both both:** + HMAC → Token → Seq → Recv Count → Recv Window → Recv Wall → Recv Mono → + Send Wall → Send Mono. Packet = 76 bytes. ✓ + +**Conclusion:** The field ordering in Section 8.1.3 is correct. This is a +**normative requirement** — incorrect ordering breaks interoperability. + +--- + +### 19.3 Server Close During Test — NOT DIRECTLY TESTED + +**Limitation:** Triggering a server-initiated close mid-test requires +specific server conditions that are difficult to set up reliably in a +black-box test. The spec's description (server MAY set Close flag on echo +reply) has not been verified externally. Reclassified as **open / +verification required** — the spec now recommends defensive handling +without asserting that this behavior occurs. + +--- + +### 19.6 Send Timestamp Capture Timing — NOT TESTABLE + +**Limitation:** The exact timing of client-side timestamp capture relative +to the send syscall is an implementation-internal detail that cannot be +verified through black-box testing. Remains an implementation choice. + +--- + +### 19.12 RTT When Server Processing > Raw Round-Trip — NOT DIRECTLY TESTED + +**Limitation:** This edge case requires server processing time to exceed +the raw client-measured RTT, which is extremely unlikely on localhost. Would +require a heavily loaded server or deliberate delays. Remains an +**open question**. + +--- + +## Additional Findings + +### Finding A: Packet Count Formula Correction + +**Spec claim:** Expected packet count = `floor(d / i) + 1` + +**Test setup:** Multiple sessions with various duration/interval combinations. + +**Command:** `irtt client -d -i 127.0.0.1 -q` + +**Results:** + +| Duration | Interval | Observed Packets | floor(d/i)+1 | ceil(d/i) | +|----------|----------|-----------------|-------------|-----------| +| 10s | 200ms | 50 | 51 | 50 | +| 1s | 200ms | 5 | 6 | 5 | +| 1s | 333ms | 4 | 4 | 4 | +| 1s | 500ms | 2 | 3 | 2 | +| 1s | 1s | 1 | 2 | 1 | +| 2s | 1s | 2 | 3 | 2 | +| 500ms | 100ms | 5 | 6 | 5 | +| 100ms | 10ms | 10 | 11 | 10 | +| 999ms | 1s | 1 | 1 | 1 | +| 1001ms | 1s | 2 | 2 | 2 | + +**Analysis:** When duration is an exact multiple of interval, the packet at +exactly `start + duration` is NOT sent (exclusive end). The correct formula +is: + +``` +expected_packets = ceil(duration / interval) +``` + +Or equivalently: +``` +if duration % interval == 0: + expected_packets = duration / interval +else: + expected_packets = floor(duration / interval) + 1 +``` + +**Conclusion:** The spec's formula `floor(d / i) + 1` is incorrect. The +correct formula is `ceil(d / i)`. The spec MUST be updated. + +--- + +### Finding B: DSCP Parameter Encodes TOS Byte Value + +**Spec claim:** DSCP valid values are "0-63 (6-bit DSCP field)" + +**Test setup:** Captured packets with various --dscp values. + +**Results:** + +| CLI Option | TOS Byte in IP Header | DSCP (6-bit) | Param Value Encoded | +|------------|----------------------|-------------|-------------------| +| --dscp=0xb8 | 0xb8 | 46 (EF) | 184 | +| --dscp=0x2e | 0x2e | 11 | 46 | +| --dscp=46 | 0x2e | 11 | 46 | +| --dscp=8 | 0x08 | 2 | 8 | +| --dscp=0x20 | 0x20 | 8 (CS1) | 32 | + +**Conclusion:** The DSCP parameter encodes the raw value as specified on the +CLI, which is used directly as the TOS/Traffic Class byte in the IP header. +The range is 0-255, not 0-63. The spec MUST be updated to reflect that +the parameter is the TOS byte value, not the 6-bit DSCP field. + +--- + +### Finding C: DSCP Not Applied to Open/Close Packets + +**Observation:** In all captures, the IP TOS field for open request, open +reply, and close request packets is always 0x00, regardless of the +negotiated DSCP value. Only echo request and echo reply packets carry the +negotiated DSCP/TOS value. + +**Verified with:** `captures/dscp-session.pcapng` + +**Conclusion:** This is **observed behavior**. It may not be a strict +protocol requirement, but implementations should be aware that DSCP is only +applied during the active test phase. + +--- + +### Finding D: No-Test Mode Packet Exchange + +**Test setup:** Captured a no-test session. + +**Command:** `irtt client -n 127.0.0.1:2113` + +**Capture:** `captures/no-test.pcapng` + +**Observed:** + +Open request (frame 1): +``` +14 a7 5b 05 [params...] +``` +- Flags: `05` = Open (0x01) | Close (0x04) +- No connection token + +Open reply (frame 2): +``` +14 a7 5b 07 00 00 00 00 00 00 00 00 [params...] +``` +- Flags: `07` = Open (0x01) | Reply (0x02) | Close (0x04) +- Connection token: `00 00 00 00 00 00 00 00` (zero) + +**Conclusion:** In no-test mode: +- Client sets both Open and Close flags in the open request. +- Server replies with Open|Reply|Close and a **zero** connection token. +- The client treats this as successful completion, not rejection. +- The Close flag in the open reply is **not** a rejection when the client + originally requested Close. + +The spec's description of "An open reply with the close flag set: the +server rejected the session" needs clarification. The Close flag in the +reply indicates rejection ONLY when the client did not request Close in its +open request. + +--- + +### Finding E: Test Vector Size Correction + +The test vectors in Section 18 are described as "target length 256 bytes +(shown truncated)." Independent HMAC verification confirms that both test +vectors (18.1 and 18.2) were computed over 92-byte packets (76-byte header ++ 16-byte payload), not 256-byte packets. The "256 bytes" description is +incorrect. + +--- + +### Finding F: Default Server Fill Pattern + +**Observation:** When no server fill is requested, the server fills reply +payloads with the repeating hex pattern `69 72 74 74` (ASCII "irtt"). This +is the default server fill as documented in server help: +`--fill=fill payload fill if not requested (default pattern:69727474)`. + +--- + +### Finding G: Close Packet Minimal Format + +**Observed close packet:** +``` +14 a7 5b 04 [8-byte connection token] +``` +Total: 12 bytes. Contains only magic (3) + flags (1) + connection token (8). +No sequence number, no payload. + +With HMAC: +``` +14 a7 5b 0c [16-byte HMAC] [8-byte connection token] +``` +Total: 28 bytes. + +--- + +### Finding H: Unreachable Server Behavior + +**Command:** `irtt client --timeouts=200ms,200ms 127.0.0.1:2121` (no server) + +**Observed:** `Error: read udp4 ... connection refused` (exit 1) + +On localhost, the OS delivers ICMP port unreachable immediately. On remote +hosts, the behavior would be a timeout after all retransmissions are +exhausted. + +--- + +### Finding I: Parameter Restriction Behavior + +**Commands:** +``` +irtt server -b 127.0.0.1 -l 100 # max length 100 +irtt client -l 200 127.0.0.1 # request length 200 +``` + +**Strict mode (default):** `[ServerRestriction] server reduced length from +200 to 100` — exit code 1. + +**Loose mode (--loose):** `[ServerRestriction] server reduced length from +200 to 100` — continues with length 100, exit code 0. + +--- + +## Captures Created + +| File | Contents | +|------|----------| +| `captures/full-session.pcapng` | Complete session: open, 3 echoes, close | +| `captures/basic-session.pcapng` | Basic session (partial, missed open) | +| `captures/no-test.pcapng` | No-test mode: open+close only | +| `captures/hmac-session.pcapng` | HMAC authenticated session | +| `captures/tstamp-none.pcapng` | Multiple sessions with different timestamp modes | +| `captures/dscp-session.pcapng` | DSCP=0xb8 session | +| `captures/dscp-46.pcapng` | DSCP=46 and DSCP=0x2e sessions | +| `captures/dscp-values.pcapng` | DSCP=8 and DSCP=0x20 sessions | +| `captures/large-packet.pcapng` | 1472-byte packet session | +| `captures/long-sfill.pcapng` | Session with long server fill string | + +## Test Vectors Created + +See `test-vectors/` directory. + +## Summary + +### Resolved Open Questions +- **19.1** Open request field layout (no token) +- **19.2** Minimum packet length calculation +- **19.4** Varint encoding compatibility +- **19.5** HMAC computation scope +- **19.7** Received window validity (bit 0 sentinel) +- **19.8** HMAC mismatch behavior +- **19.11** Field ordering verification + +### Partially Resolved +- **19.9** Minimum restricted interval safety floor (strict mode rejects, + but exact floor for decreased intervals untested) +- **19.10** Maximum parameter buffer size (observed well under 128 bytes) + +### Still Unresolved +- **19.3** Server close during test (hard to trigger) +- **19.6** Send timestamp capture timing (implementation detail, not testable) +- **19.12** RTT when server processing exceeds raw RTT (hard to trigger) + +### Spec Corrections Applied +1. Packet count formula: `ceil(d/i)` not `floor(d/i)+1` +2. DSCP parameter: TOS byte value (0-255) not 6-bit DSCP (0-63) +3. Test vector size: 92 bytes not 256 bytes +4. No-test mode close flag: clarified in Sections 6.2, 8.3, and 14 +5. Server close during test (19.3): reclassified as open/verification + required; removed source-derived claims; spec now recommends + defensive handling without asserting the behavior occurs +6. Received window late-packet behavior: removed unverified claim about + late packets producing window=0; retained verified bit-0 sentinel diff --git a/docs/CLEANROOM_NOTES.md b/docs/CLEANROOM_NOTES.md new file mode 100644 index 0000000..d99f407 --- /dev/null +++ b/docs/CLEANROOM_NOTES.md @@ -0,0 +1,189 @@ +# Clean-Room Implementation Notes + +## Purpose + +This document records the clean-room boundary for the IRTT client +reimplementation project. The goal is to ensure that the implementation +is developed from a behavioral/protocol specification only, without exposure +to the GPL-licensed upstream source code. + +## Clean-Room Boundary + +- **Contaminated side:** The specification agent (this side) inspected the + upstream source code, documentation, tests, CLI behavior, and release + notes to understand the IRTT protocol. + +- **Clean side:** The implementation agent will receive only the + `IRTT_CLIENT_PROTOCOL_SPEC.md` document. The implementation agent MUST NOT + see the upstream source code. + +## Source Materials Inspected + +The following materials from the upstream IRTT repository (heistp/irtt, +GPLv2) were inspected to produce the protocol specification: + +1. **Documentation files:** + - `README.md` — project overview, FAQ, features, limitations + - `doc/irtt.md` — main man page + - `doc/irtt-client.md` — client man page with CLI options and JSON output format + - `doc/irtt-server.md` — server man page (referenced, not fully read) + - `CHANGES.md` — changelog + +2. **Source files (inspected for protocol behavior only):** + - Protocol version and constants + - Packet format, field layout, magic bytes, flags + - Parameter serialization format (tag-value pairs, varint encoding) + - Connection lifecycle (open, echo, close handshakes) + - Client configuration and defaults + - Server connection handling and parameter restriction logic + - HMAC computation + - Timestamp handling and clock types + - Received stats (count and window) format + - Round-trip data recording and measurement calculations + - Result computation (RTT, OWD, IPDV, loss) + - Error codes and error handling + - Packet test vectors (serialized bytes for known inputs) + - Network layer (IP version, DSCP, socket options) + - Wait/drain behavior + +3. **Test files:** + - Packet serialization tests (used to derive test vectors only; test + code itself was not copied) + +## Statement of Non-Copying + +No upstream source code, comments, or pseudocode were copied into the +specification document (`IRTT_CLIENT_PROTOCOL_SPEC.md`). The specification +describes only externally observable protocol behavior, wire format, and +interoperability requirements. + +Test vectors in the specification were derived from the known inputs and +expected byte sequences found in the upstream test suite. Only the input +parameters and resulting byte arrays were used; no test logic or code was +copied. + +## Post-Drafting Audit + +After the initial specification was drafted, a clean-room audit was +performed. The following issues were identified and corrected: + +1. **Upstream private type names leaked:** "ctoken" and "seqno" appeared + as parenthetical abbreviations in the terminology table. Removed. + +2. **Upstream implementation preference disclosed:** A note about IPv4 + preference in dual-stack resolution revealed an upstream implementation + choice. Removed. + +3. **Go-specific function references:** References to `binary.PutUvarint`, + `binary.PutVarint`, and `encoding/binary` package named the upstream + language's standard library. Replaced with language-neutral descriptions + (LEB128, protobuf-style zigzag). + +4. **Internal scheduling algorithm described:** The send scheduling section + described a specific "snap-to-interval" drift-correction algorithm that + mirrored upstream control flow. Replaced with observable behavior + description (packets sent at approximately ideal times, drift + compensation is implementation-defined). + +5. **Upstream timer algorithm named:** "The upstream uses exponential + averaging of timer errors" revealed an internal implementation choice. + Removed. + +6. **RTT edge case speculated:** The spec asserted that server processing + time exceeding raw RTT should cause subtraction to be skipped — a guard + the upstream does not actually implement. Moved to Open Questions. + +7. **Received window validity asserted normatively:** The behavior of + window=0 as an invalidity sentinel was stated as fact when it should be + verification-required. Softened with reference to Open Questions. + +8. **Go concurrency term leaked:** "goroutine" appeared in the result model + section. Replaced with "operation." + +9. **Upstream test design referenced:** Section 18.2 described what "the + upstream test" was testing internally. Rewritten to describe the test + vector neutrally. + +10. **Upstream constants asserted normatively:** The minimum restricted + interval (1 second) and maximum parameter buffer size (128 bytes) are + internal constants that were stated as protocol requirements. Moved to + Open Questions with verification suggestions. + +11. **Timestamp capture order stated as MUST:** Implementation-level timing + advice was stated as a protocol requirement. Changed to SHOULD with + measurement-accuracy rationale. + +12. **Internal recording structure described:** Section 19.6 described a + "pre-send timestamp" used for send-call-time measurement, which is an + internal recording detail. Rewritten to focus on observable behavior. + +13. **Self-contradictory correction note:** Section 8.2 contained an inline + "Correction:" note that was confusing and revealed the drafting process. + Cleaned up. + +14. **Field ordering lacked verification note:** The critical field order + in Section 8.1.3 was derived from source and stated without any + verification caveat. Added verification note and new Open Question 19.11. + +## Clean-Room Compliance Checklist + +- [x] No upstream source code included. +- [x] No upstream comments quoted. +- [x] No upstream private function names included. +- [x] No upstream private type names included. +- [x] No upstream file/module layout described. +- [x] No pseudocode derived from upstream implementation. +- [x] No implementation-language API design included. +- [x] No Go-specific language references remain. +- [x] No upstream implementation preferences or algorithm choices disclosed. +- [x] Protocol behavior described independently. +- [x] Source-derived constants marked for verification. +- [x] Ambiguous behavior marked for verification (Section 19). +- [x] Black-box tests proposed where needed (Section 17). +- [x] Post-drafting audit completed and documented (this section). +- [x] Second-pass scrub completed (see below). + +## Second-Pass Clean-Room Scrub + +A second independent review was performed on 2026-04-28 by an agent that +had not seen the upstream source. The following issues were identified and +corrected: + +1. **Implementation architecture directive in scope:** Section 2 stated + "Library-first implementation target," which prescribes internal + architecture. Removed; the spec now says only that CLI design is not + prescribed. + +2. **Concurrency architecture leaked:** Section 6.3 described "a separate + receive path processes incoming reply packets concurrently," which + prescribes an internal threading/task model. Rewritten to state the + observable requirement (send and receive occur concurrently) without + prescribing how. + +3. **Source-derived design rationale in echo request:** Section 8.4 + explained that zeroed fields exist to "use the same field layout that + appears in echo replies," which reveals internal design reasoning. + Simplified to state only the observable purpose (reaching the + negotiated packet length). + +4. **Source-exhaustive knowledge parenthetical:** Section 8.6 said + "(currently only ServerFill)" for string-typed values, revealing + knowledge of the complete parameter set from source inspection. + Removed the parenthetical. + +5. **Upstream CLI syntax leaked:** Section 10.10 used `NxD`/`NrD`/`D` + notation for wait variants, which mirrors upstream CLI option syntax. + Replaced with plain-language descriptions. + +6. **Algorithm name disclosed:** Section 12.9 named "Welford's online + algorithm" as a MAY recommendation. This is implementation guidance. + Replaced with "the method of computing running statistics is + implementation-defined." + +7. **Implementation-specific error category:** Section 14 listed + "Allocate results buffer failure," implying a pre-allocated buffer + strategy. Generalized to "Insufficient resources for test parameters." + +8. **Language-specific references in CLEANROOM_NOTES.md:** The Purpose + section named "the Rust implementation" and the checklist mentioned + "No Rust API design." Both replaced with language-neutral phrasing. diff --git a/docs/IMPLEMENTATION_PLAN.md b/docs/IMPLEMENTATION_PLAN.md new file mode 100644 index 0000000..bdd0a28 --- /dev/null +++ b/docs/IMPLEMENTATION_PLAN.md @@ -0,0 +1,2036 @@ +# IRTT-RS Implementation Plan + +## 1. Inputs Reviewed + +### RUST_DESIGN.md + +**Contribution:** Defines the full Rust architecture: four-crate workspace (`irtt-proto`, `irtt-client`, `irtt-stats`, `irtt-cli`), the `Client` / `ManagedClient` split, event model, CLI output formats, dependency choices, and milestone sequence. + +**Sufficiency:** Sufficient for implementation planning. The design is thorough and internally consistent. A few minor amendments are noted in Section 18. + +**Inconsistencies found:** +- None that would block implementation. The design is well-aligned with the protocol spec and black-box findings. + +### IRTT_CLIENT_PROTOCOL_SPEC.md + +**Contribution:** Normative protocol specification. Defines wire format, field ordering, parameter serialization (varint/zigzag), session lifecycle, timing semantics, measurement formulas, validation rules, and error handling. Includes two packet test vectors with HMAC. + +**Sufficiency:** Sufficient for implementing a conforming client. All resolved open questions have clear answers. Three open questions remain (19.3, 19.6, 19.12) — none block initial implementation. + +**Inconsistencies found:** +- The spec's packet count formula was corrected to `ceil(d/i)` — the spec itself already reflects this correction. +- DSCP parameter range was corrected to 0–255 (TOS byte) — the spec already reflects this. +- Test vector sizes corrected to 92 bytes — the spec already reflects this. +- The flags byte in test vector 18.2 is `0x08` (HMAC only, no Reply flag). The spec explicitly notes this is for HMAC/layout verification only and that real packets require Reply flag `0x02`. This is clear and not a problem. + +### BLACKBOX_VERIFICATION_REPORT.md + +**Contribution:** Independent black-box verification of protocol behavior against irtt 0.9.1. Resolves 7 open questions, partially resolves 2, confirms spec corrections, and provides packet captures as ground truth. + +**Sufficiency:** Sufficient. The remaining unresolved items (19.3, 19.6, 19.12) are edge cases with clear defensive strategies. + +**Inconsistencies found:** +- None with the spec (corrections already applied). + +### CLEANROOM_NOTES.md + +**Contribution:** Documents the clean-room boundary, source materials inspected by the specification agent, post-drafting audit, and second-pass scrub. + +**Sufficiency:** Sufficient. Clear separation between contaminated (spec-writing) and clean (implementation) sides. + +**Inconsistencies found:** +- None. + +### Clean Packet Test Vectors (`docs/test-vectors.md`) + +**Contribution:** 12 packet test vectors from actual network captures against irtt 0.9.1. Covers: open request/reply (with and without HMAC), echo request/reply (with and without HMAC, various stats/timestamp configs), close request (with and without HMAC), no-test mode request/reply, minimal echo packet (16 bytes), and midpoint timestamp echo reply. + +**Sufficiency:** Excellent. Combined with the two HMAC test vectors in the protocol spec (Section 18.1, 18.2), this provides 14 total test vectors covering every packet type and major field combination. Sufficient for thorough protocol validation. + +**Inconsistencies found:** +- None. All vectors are consistent with the protocol spec and black-box report. + +--- + +## 2. Executive Summary + +- **Crate structure:** Four crates in a Cargo workspace — `irtt-proto` (wire protocol), `irtt-client` (socket + session + events), `irtt-stats` (event aggregation), `irtt-cli` (user-facing binary). +- **Manual vs managed client:** `Client` is the caller-paced, no-thread, no-stats core. `ManagedClient` is a threaded convenience runner built on the same primitives. Both live in `irtt-client`. +- **Event-first design:** `irtt-client` emits `ClientEvent` values. It never aggregates statistics. Consumers decide what to do with events. +- **Stats separation:** `irtt-stats` consumes `ClientEvent` and provides cumulative, rolling, and summary statistics. It is an optional dependency of `irtt-cli`. +- **CLI output philosophy:** Line-oriented, awk-friendly, delimiter-configurable output. Human, machine, simple, rtt-us, and optional JSONL formats. Default builds include stats; minimal/router builds can omit them. +- **Interop testing strategy:** `cargo test` runs pure unit tests without external dependencies. `cargo test --features interop` requires `irtt` server in PATH. `tshark` is diagnostic-only, never a CI dependency. +- **Clean-room compliance:** Implementation works only from the four clean artifacts. No upstream source, tests, or contaminated notes. +- **Router friendliness:** No mandatory async runtime. No busy-spin by default. Feature-gated optional capabilities. Minimal binary possible with `--no-default-features`. +- **First-class consumers:** `sqm-autorate-rust` (manual `Client` API), `cake-autorate` (CLI machine output), general users (CLI with stats). +- **Staged implementation:** 10 milestones, each independently testable, building from protocol layer up. + +--- + +## 3. Architecture Confirmation + +The intended architecture from `RUST_DESIGN.md` is confirmed with no structural changes needed. + +### `irtt-proto` + +**Public responsibility:** +- Binary wire format encoding and decoding +- Packet constants (magic, flags, protocol version) +- Varint/zigzag encoding and decoding +- Parameter model and serialization +- Packet layout calculation (minimum packet size for given parameters) +- Open request/reply encoding and decoding +- Echo request encoding, echo reply decoding +- Close request encoding +- HMAC-MD5 calculation and verification + +**Forbidden responsibility:** +- Sockets, threads, channels, timers +- Statistics aggregation +- Session runtime control +- Event fanout +- ECN/DSCP/TOS socket metadata (these are IP-layer, not IRTT payload) + +**Main public types:** +- `Magic`, `Flags`, `ProtocolVersion` +- `Params`, `ParamTag`, `ReceivedStats`, `StampAt`, `Clock`, `ServerFill` +- `OpenRequest`, `OpenReply` +- `EchoRequest`, `EchoReply` +- `CloseRequest` +- `PacketLayout` +- `HmacKey`, HMAC computation functions +- `ProtoError` + +**Internal modules:** +- `varint` — uvarint/zigzag encode/decode +- `params` — parameter serialization/deserialization +- `header` — magic/flags/field parsing +- `hmac` — HMAC-MD5 wrapper +- `layout` — packet size calculation + +**Feature flags:** +- `serde` — derive Serialize/Deserialize on public types + +**Dependencies:** +- `thiserror`, `hmac`, `md-5`, `subtle`, `fastrand` +- `serde` (optional) + +### `irtt-client` + +**Public responsibility:** +- Connected UDP socket setup and configuration via `socket2` +- Session negotiation (open/close lifecycle) +- Caller-paced manual client API (`Client`) +- Optional managed threaded runner (`ManagedClient`) +- Probe sending and reply receiving +- Packet validation +- Timestamp capture +- RTT sample construction +- Duplicate/late/loss detection +- Public event types (`ClientEvent`) +- Event fanout for managed mode + +**Forbidden responsibility:** +- Aggregate statistics (min/max/mean/median/stddev/variance) +- Rolling windows +- Human-readable reports +- CLI formatting +- Mandatory full sample history + +**Main public types:** +- `ClientConfig`, `SocketConfig` +- `Client`, `SessionState` +- `ManagedClient`, `ManagedClientSession` +- `EventHub`, `EventSubscription`, `SubscriberConfig`, `SubscriberOverflow` +- `ClientEvent`, `RttSample`, `OneWayDelaySample`, `ServerTiming` +- `ReceivedStatsSample`, `PacketMeta`, `LossKind` +- `ClientWarning`, `SessionOutcome`, `SessionEndReason` +- `NegotiatedParams`, `NegotiationPolicy`, `ValidationPolicy` +- `RunMode`, `LossDetection`, `TimerMode` +- `ReconnectPolicy`, `CancellationToken` +- `ClientError` +- `ClientTimestamp` + +**Internal modules:** +- `socket` — `ClientSocket` wrapper +- `session` — session state machine +- `probe` — pending probe tracking +- `validate` — packet validation logic +- `timing` — timestamp capture and RTT computation +- `managed` — managed runner threads and event hub + +**Feature flags:** +- `serde` — derive Serialize/Deserialize on public event types +- `tracing` — structured logging +- `ancillary` — platform control-message APIs for ECN/DSCP/kernel RX timestamps + +**Dependencies:** +- `irtt-proto`, `thiserror`, `socket2`, `flume`, `fastrand` +- `serde`, `tracing` (optional) +- `rustix` or similar (optional, for ancillary) + +### `irtt-stats` + +**Public responsibility:** +- Consume `ClientEvent` values +- Cumulative statistics +- Rolling time-window statistics +- Rolling sample-count statistics +- Finite-run summaries +- RTT/IPDV/OWD/server-processing/loss statistics + +**Forbidden responsibility:** +- Sockets, packet encoding/decoding +- Session negotiation +- Thread/runtime ownership +- CLI parsing + +**Main public types:** +- `StatsCollector` (or `StatsEngine`) +- `CumulativeStats` +- `RollingStats` +- `FiniteSummary` +- `RttStats`, `IpdvStats`, `LossStats` +- `StatsConfig` + +**Internal modules:** +- `running` — online mean/variance computation +- `rolling` — windowed statistics +- `summary` — finite-run summary builder +- `median` — median computation strategy + +**Feature flags:** +- `serde` — derive Serialize/Deserialize on stats types + +**Dependencies:** +- `irtt-client` (for `ClientEvent` types) +- `thiserror` +- `serde` (optional) + +### `irtt-cli` + +**Public responsibility:** +- User-facing executable +- Command-line argument parsing +- Output formatting (human, machine, simple, rtt-us, jsonl) +- Stats display (when feature-enabled) +- Signal handling (Ctrl-C) +- Exit codes + +**Forbidden responsibility:** +- Protocol reimplementation +- Session logic reimplementation +- Statistics reimplementation + +**Main public types:** +- (binary crate — minimal public API) +- Internal: `CliConfig`, `OutputFormatter`, `MachineFormatter`, `SimpleFormatter`, `RttUsFormatter`, `HumanFormatter` + +**Internal modules:** +- `args` — clap argument parsing +- `format` — output formatting +- `run` — test execution orchestration +- `signal` — Ctrl-C handling + +**Feature flags:** +- `default = ["stats"]` +- `stats = ["dep:irtt-stats"]` +- `json = ["dep:serde_json", "irtt-client/serde", "irtt-stats?/serde"]` +- `tracing = ["irtt-client/tracing"]` +- `ancillary = ["irtt-client/ancillary"]` +- `interop = []` + +**Dependencies:** +- `irtt-client`, `irtt-stats` (optional), `clap`, `thiserror`, `ctrlc` +- `serde_json` (optional) + +--- + +## 4. Protocol Implementation Plan + +### Constants + +``` +MAGIC: [0x14, 0xA7, 0x5B] +PROTOCOL_VERSION: 1 +FLAG_OPEN: 0x01 +FLAG_REPLY: 0x02 +FLAG_CLOSE: 0x04 +FLAG_HMAC: 0x08 +RESERVED_FLAGS_MASK: 0xF0 +HMAC_SIZE: 16 +TOKEN_SIZE: 8 +SEQ_SIZE: 4 +RECV_COUNT_SIZE: 4 +RECV_WINDOW_SIZE: 8 +TIMESTAMP_SIZE: 8 +``` + +### Flags + +Implement `Flags` as a newtype over `u8` with named bit accessors: +- `is_open()`, `is_reply()`, `is_close()`, `is_hmac()` +- `has_reserved_bits()` — returns true if bits 4–7 are set +- Builder pattern for constructing outgoing flags + +### Field Layout + +Implement `PacketLayout` that computes field offsets and total minimum size given: +- Whether HMAC is present +- Whether this is an open request (no token), echo (token + seq + optional fields), or close (token only) +- `ReceivedStats` mode +- `StampAt` mode +- `Clock` mode + +The layout must match the field ordering verified in BLACKBOX_VERIFICATION_REPORT Section 19.11: +1. HMAC (if flag set) +2. Connection Token (except open request from client) +3. Sequence Number (echo only) +4. Received Count (if count or both) +5. Received Window (if window or both) +6. Receive Wall (if receive or both, and wall or both clock) +7. Receive Mono (if receive or both, and mono or both clock) +8. Midpoint Wall (if midpoint, and wall or both clock) — mutually exclusive with recv/send +9. Midpoint Mono (if midpoint, and mono or both clock) +10. Send Wall (if send or both, and wall or both clock) +11. Send Mono (if send or both, and mono or both clock) + +### Endianness + +All multi-byte integers are little-endian. Use `u32::from_le_bytes()`, `u64::from_le_bytes()`, `i64::from_le_bytes()` for reading. Use `.to_le_bytes()` for writing. + +### Varint / Zigzag + +Implement two functions: +- `encode_uvarint(value: u64, buf: &mut [u8]) -> usize` — standard unsigned LEB128 +- `decode_uvarint(buf: &[u8]) -> Result<(u64, usize), ProtoError>` — returns (value, bytes_consumed) + +And for signed: +- `encode_varint(value: i64, buf: &mut [u8]) -> usize` — zigzag then LEB128 +- `decode_varint(buf: &[u8]) -> Result<(i64, usize), ProtoError>` — LEB128 then unzigzag + +Zigzag transform: +- Encode: `((v << 1) ^ (v >> 63)) as u64` +- Decode: `((uv >> 1) as i64) ^ -((uv & 1) as i64)` + +### Parameter Encoding/Decoding + +Parameters are tag-value pairs: +- Tag: uvarint +- Value: varint (signed zigzag), except for string-typed values +- String values: uvarint length prefix + raw UTF-8 bytes + +Define `Params` struct: +``` +protocol_version: i64 +duration_ns: i64 +interval_ns: i64 +length: i64 +received_stats: ReceivedStats +stamp_at: StampAt +clock: Clock +dscp: i64 +server_fill: Option +``` + +Serialize: iterate fields, skip zero values (except protocol_version which is always 1), write tag + value. + +Deserialize: read tag-value pairs in a loop, match known tags, silently ignore unknown tags. + +Verified encoding examples from BLACKBOX_VERIFICATION_REPORT: +- ProtocolVersion=1: tag `01`, value zigzag(1)=2 → `02` +- Duration=3s: tag `02`, value zigzag(3000000000)=6000000000 → `80 f8 82 ad 16` +- DSCP=0xb8: tag `08`, value zigzag(184)=368 → `f0 02` + +### Open Request/Reply + +**Open Request encoding:** +1. Write magic (3 bytes) +2. Write flags (1 byte): Open | optional HMAC | optional Close (no-test) +3. If HMAC: write 16 zero bytes (placeholder) +4. No connection token +5. Write serialized params as payload +6. If HMAC: compute HMAC-MD5 over entire buffer with HMAC field zeroed, write result into HMAC field + +**Open Reply decoding:** +1. Validate magic +2. Read flags — must have Open | Reply. Check for Close flag and reserved bits. +3. If HMAC flag: extract HMAC, verify +4. Read connection token (8 bytes) +5. Parse params from remaining payload +6. Determine outcome: normal open, no-test ack, or rejection + +### Echo Request/Reply + +**Echo Request encoding:** +1. Write magic + flags (0x00 or 0x08 for HMAC) +2. If HMAC: write 16 zero bytes +3. Write connection token (8 bytes) +4. Write sequence number (4 bytes LE) +5. Write zeroed received stats fields (if negotiated) +6. Write zeroed timestamp fields (if negotiated) +7. Write payload to reach negotiated length +8. If HMAC: compute and write + +**Echo Reply decoding:** +1. Validate magic +2. Read flags — must have Reply. Check HMAC, Close, reserved bits. +3. If HMAC: extract and verify +4. Read connection token +5. Read sequence number +6. Read received count (if negotiated) +7. Read received window (if negotiated) +8. Read timestamp fields (per StampAt + Clock) +9. Remaining bytes are payload (ignore for measurement purposes) + +### Close Request + +**Encoding:** +1. Write magic + flags (Close | optional HMAC) +2. If HMAC: write 16 zero bytes +3. Write connection token +4. If HMAC: compute and write + +Total: 12 bytes without HMAC, 28 bytes with HMAC. + +### HMAC-MD5 Handling + +1. Use `hmac` and `md-5` crates +2. Compute over entire packet buffer with HMAC field (bytes 4–19) zeroed +3. Use `subtle::ConstantTimeEq` for verification +4. HMAC key is raw bytes (from user-provided string, typically UTF-8) + +### Packet Vector Tests + +Implement tests for: +1. Test vector 18.1 (echo request with HMAC, 92 bytes) — verify HMAC, field layout, field values +2. Test vector 18.2 (echo reply with HMAC, 92 bytes) — verify HMAC, field layout, field values +3. Varint encoding/decoding round-trips for all verified values from BLACKBOX_VERIFICATION_REPORT +4. Parameter encoding/decoding round-trips +5. Layout calculation for all 6 verified field combinations from Section 19.11 +6. Flags construction and parsing +7. Open request encoding (verify no token field present) +8. Close request encoding (verify 12/28 byte sizes) + +--- + +## 5. Client Manual API Plan + +### `ClientConfig` + +``` +server_addr: String // hostname:port or IP:port +duration: Option // None for continuous +interval: Duration // default 1s +length: u32 // 0 = minimum +received_stats: ReceivedStats // default Both +stamp_at: StampAt // default Both +clock: Clock // default Both +dscp: u8 // default 0 +hmac_key: Option> +server_fill: Option +open_timeouts: Vec // default [1s, 2s, 4s, 8s] +run_mode: RunMode +negotiation_policy: NegotiationPolicy +validation_policy: ValidationPolicy +loss_detection: LossDetection +timer_mode: TimerMode +socket_config: SocketConfig +``` + +### `SocketConfig` + +``` +bind_addr: Option +ttl: Option +ipv4_only: bool +ipv6_only: bool +recv_timeout: Option // for blocking recv in manual mode +``` + +### `ClientSocket` + +Internal struct wrapping `std::net::UdpSocket` (created via `socket2::Socket`). + +Responsibilities: +- Create socket (IPv4 or IPv6 based on resolved address) +- Bind to local address (or ephemeral) +- Set DSCP/TOS via `socket2` (`set_tos` for IPv4, `set_traffic_class_v6` for IPv6) — applied only for echo phase per Finding C +- Set TTL/hop limit if configured +- Connect to resolved server address +- `send(&self, buf: &[u8]) -> io::Result` +- `recv(&self, buf: &mut [u8]) -> io::Result` +- `set_read_timeout(&self, timeout: Option)` +- `try_clone(&self) -> io::Result` — for managed mode + +MVP `PacketMeta` returns all `None` fields. Future `ancillary` feature fills them. + +### `SessionState` + +Internal struct tracking: +- `connection_token: u64` +- `negotiated_params: NegotiatedParams` +- `next_seq: u32` (wire sequence) +- `logical_seq: u64` +- `packets_sent: u64` +- `replies_received: u64` +- `duplicates: u64` +- `late_count: u64` +- `malformed: u64` +- `max_received_seq: Option` (for late detection) +- `pending_probes: BoundedPendingMap` (seq → PendingProbe) +- `start_time: Option` +- `end_time: Option` (computed from start + duration) +- `phase: SessionPhase` (Opening, Active, Draining, Closing, Completed) + +### `PendingProbe` + +``` +logical_seq: u64 +wire_seq: u32 +sent_at_mono: Instant +sent_at_wall: SystemTime +replied: bool +``` + +### `BoundedPendingMap` + +A bounded map from `u32` (wire seq) to `PendingProbe`. Capacity limited to prevent unbounded growth in continuous mode. Oldest entries evicted when capacity reached (these become loss events). + +### `Client::connect` + +1. Resolve server address (DNS lookup if hostname) +2. Create `ClientSocket` (socket2 → std) +3. Bind local address +4. Connect UDP socket +5. Return `Client` in `Opening` phase + +### `Client::open` + +1. Encode open request with proposed params +2. Send open request +3. Set read timeout to first timeout value +4. Loop: recv → validate magic/flags/HMAC → parse open reply +5. On timeout: retransmit with next timeout value +6. On all timeouts exhausted: return `ClientError::OpenTimeout` +7. On valid reply: + - If Close flag and client didn't request close: return `ClientError::ServerRejected` + - If Close flag and client requested close (no-test): return `SessionEnded` event + - Extract connection token, negotiated params + - Apply negotiation policy (reject or accept restrictions) + - Compute minimum packet length from negotiated params + - Transition to Active phase + - Set DSCP on socket (only now, for echo phase) + - Return `SessionStarted` event + +### `Client::send_probe` + +1. Assert Active phase +2. Check if duration exceeded (if finite mode) → if so, transition to Draining +3. Encode echo request with current sequence number, connection token +4. Call `socket.send()` +5. Capture send timestamp (Instant + SystemTime) immediately after send returns +6. Insert PendingProbe into pending map +7. Increment sequence counters +8. Return any events (e.g., eviction-triggered loss events from bounded map) + +### `Client::recv_once` + +1. Attempt one `socket.recv()` (may block up to read timeout, or nonblocking) +2. Capture receive timestamp immediately after recv returns +3. Validate packet (magic, flags, reserved bits, HMAC, connection token, sequence) +4. Match to pending probe +5. If duplicate: emit `DuplicateReply`, do not update original +6. If late (seq < max_received_seq): emit `LateReply` +7. Compute `RttSample` (raw, server_processing, adjusted) +8. Compute `OneWayDelaySample` if wall timestamps available +9. Extract `ReceivedStatsSample` if negotiated +10. Emit `EchoReply` event +11. Return `Ok(Some(events))` or `Ok(None)` on timeout/would-block + +### `Client::recv_available` + +Loop calling `recv_once` up to a budget limit (max packets or max duration). Collect all events. + +### `Client::poll_timeouts` + +1. Check all pending probes against loss detection policy +2. For probes exceeding timeout: emit `EchoLoss` event, remove from pending map +3. Use received window from most recent reply to classify loss direction where possible +4. Return collected events + +### `Client::close` + +1. Reset DSCP to 0 on socket (per Finding C) +2. Encode and send close request +3. Transition to Completed phase +4. Return `SessionEnded` event with outcome + +### No-Test Mode + +`RunMode::NoTest`: `Client::open` sets both Open and Close flags. On valid reply with Close flag and zero token, immediately returns `SessionEnded`. No echo phase. + +### Continuous Mode + +`RunMode::Continuous`: No duration limit. `send_probe` never transitions to Draining on its own. Caller is responsible for stopping (by calling `close` or dropping). Bounded pending map prevents unbounded memory growth. + +### Finite Duration/Count Modes + +`RunMode::Duration(d)`: `send_probe` checks `Instant::now() >= start + d` before sending. `send_probe` returns a "test complete" indication when done. + +`RunMode::Count(n)`: `send_probe` checks `logical_seq >= n` before sending. + +### Negotiation Policy + +- `AcceptServerRestrictions`: accept any server-returned params, emit warnings if changed +- `RequireExact`: reject if any param was changed by server +- `RequireWithin(LocalRequirements)`: reject if params exceed caller-specified bounds + +### Validation Policy + +- `Strict`: abort session on any malformed packet +- `DropMalformed`: silently drop malformed packets, continue +- `DropUnrelatedAbortMalformedSession`: drop unrelated garbage, abort on malformed session packets + +### Loss Detection Policy + +- `Disabled`: never emit loss events from timeout +- `Timeout(Duration)`: emit loss if no reply within fixed duration +- `MultipleOfInterval(u32)`: emit loss if no reply within N × interval + +### Cancellation for Manual Mode + +No signal handling in `Client`. Caller controls lifecycle by: +- Not calling `send_probe` after deciding to stop +- Calling `close` for graceful shutdown +- Dropping `Client` for ungraceful shutdown + +--- + +## 6. Managed Runner Plan + +### Overview + +`ManagedClient` wraps the same socket and session primitives as `Client`, but runs them on dedicated threads with event fanout. + +### Thread Roles + +**Coordinator thread:** +- Owns session state (equivalent of `SessionState`) +- Receives compact `SentRecord` from sender via internal `flume` channel +- Receives compact `RecvRecord` from receiver via internal `flume` channel +- Correlates sent/received records +- Computes RTT samples +- Manages pending probe map +- Handles loss timeouts +- Detects duplicates and late replies +- Emits `ClientEvent` through `EventHub` +- Handles graceful shutdown and close packet + +**Sender/pacer thread:** +- Owns a cloned socket handle (write half) +- Maintains absolute send schedule: `scheduled_time(seq) = session_start + seq * interval` +- Prepares echo request packets +- Sends packets at scheduled times +- Captures send timestamps immediately after send +- Sends compact `SentRecord { wire_seq, logical_seq, sent_at_mono, sent_at_wall }` to coordinator +- Must NOT perform: event fanout, stats, subscriber delivery, blocking coordination + +**Receiver thread:** +- Owns a cloned socket handle (read half) +- Blocks on `socket.recv()` +- Captures receive timestamp immediately after recv returns, before parsing +- Parses/validates reply minimally +- Sends compact `RecvRecord { wire_seq, received_at_mono, received_at_wall, server_timestamps, received_stats, packet_meta, raw_packet_info }` to coordinator +- Must NOT perform: RTT computation, event fanout, stats + +### Internal Channels + +- Sender → Coordinator: `flume::bounded(capacity)` — capacity sized to avoid dropped sends (e.g., 256) +- Receiver → Coordinator: `flume::bounded(capacity)` — similar +- Control → All: `CancellationToken` (Arc) checked by all threads + +Internal channels must never silently drop records. If sender or receiver channel is full, this indicates coordinator is stuck — treat as fatal. + +### Event Hub / Fanout + +``` +EventHub { + subscribers: Vec +} + +SubscriberSlot { + sender: flume::Sender + config: SubscriberConfig + drops: AtomicU64 +} +``` + +When coordinator emits an event: +1. Clone event for each subscriber +2. Attempt send according to subscriber's overflow policy: + - `Block`: blocking send (only if subscriber is trusted to keep up) + - `DropNewest`: try_send, increment drops on failure + - `DropOldest`: complex — not trivially supported by flume; implement by draining one and retrying, or use a ring-buffer channel + - `Disconnect`: try_send, on failure remove subscriber + +**Critical invariant:** Public subscriber backpressure must NOT delay the sender/pacer thread. The sender never touches the event hub. Only the coordinator touches the event hub, and the coordinator is not on the timing-critical path. + +### Subscriber Configuration + +``` +SubscriberConfig { + capacity: usize // channel buffer size + overflow: SubscriberOverflow + event_filter: EventFilter // optional: subscribe only to specific event types +} +``` + +Preset profiles: +- `SubscriberConfig::stats()`: Block, large capacity, all events +- `SubscriberConfig::live()`: DropOldest, small capacity (e.g., 4), EchoReply/EchoLoss only +- `SubscriberConfig::cli()`: DropNewest, moderate capacity (e.g., 1024), all events +- `SubscriberConfig::debug()`: DropNewest, small capacity, all events + +### Shutdown / Stop / Join + +1. `session.stop()`: sets `CancellationToken`, non-blocking +2. Sender thread: detects cancellation, exits loop +3. Receiver thread: detects cancellation (via recv timeout + check), exits +4. Coordinator: detects cancellation, drains remaining records, sends close packet, emits `SessionEnded`, exits +5. `session.join(self) -> Result`: joins all threads, returns outcome + +### Thread Lifecycle + +1. `ManagedClient::start(config)`: + - Create socket, perform open handshake (on calling thread) + - On success: clone socket handles + - Spawn sender, receiver, coordinator threads + - Return `ManagedClientSession` (owns join handles + event hub + cancellation token) +2. Subscribers can be added before or after start (before start is preferred; adding after requires the hub to be shared) +3. On thread panic: coordinator detects via channel disconnect, emits FatalError, shuts down remaining threads + +### Timer Strategy + +Sender uses absolute scheduling. For `TimerMode::Normal`: +- Compute `wake_time = session_start + seq * interval` +- Sleep until `wake_time` using `std::thread::sleep` or `std::thread::park_timeout` +- After waking, immediately send + +For `TimerMode::Precise { spin_threshold }`: +- Sleep until `wake_time - spin_threshold` +- Busy-spin for remaining time +- Only for users who explicitly opt in (not default, not router-safe) + +### Reuse of Manual/Session Primitives + +The managed runner reuses: +- `irtt-proto` for all encoding/decoding +- `ClientSocket` for socket operations +- `PacketLayout` for buffer sizing +- Validation logic from the session module +- RTT/timestamp computation logic + +It does NOT reuse `Client` directly (the managed runner has different ownership semantics), but the logic is shared at the module level. + +--- + +## 7. Event Model Plan + +### `ClientEvent` + +```rust +pub enum ClientEvent { + SessionStarted { + session_id: SessionId, + remote: SocketAddr, + negotiated: NegotiatedParams, + }, + EchoReply { + session_id: SessionId, + seq: u64, + wire_seq: u32, + remote: SocketAddr, + sent_at: ClientTimestamp, + received_at: ClientTimestamp, + rtt: RttSample, + one_way: Option, + server_timing: Option, + received_stats: Option, + packet_meta: PacketMeta, + }, + EchoLoss { + session_id: SessionId, + seq: u64, + wire_seq: u32, + kind: LossKind, + }, + DuplicateReply { + session_id: SessionId, + seq: u64, + wire_seq: u32, + }, + LateReply { + session_id: SessionId, + seq: u64, + wire_seq: u32, + rtt: Option, + }, + Warning { + session_id: Option, + warning: ClientWarning, + }, + SessionEnded { + session_id: SessionId, + reason: SessionEndReason, + }, + FatalError { + session_id: Option, + error: ClientErrorKind, + }, +} +``` + +### `RttSample` + +```rust +pub struct RttSample { + pub raw: Duration, + pub adjusted: Option, + pub server_processing: Option, +} + +impl RttSample { + pub fn effective(&self) -> Duration { + self.adjusted.unwrap_or(self.raw) + } +} +``` + +**Rules:** +- `raw = client_receive_mono - client_send_mono` +- `server_processing = server_send_mono - server_receive_mono` (prefer mono, fallback wall) +- `adjusted = raw - server_processing` ONLY IF `server_processing <= raw` +- If `server_processing > raw`: `adjusted = None`, emit `ClientWarning::ServerProcessingExceedsRawRtt` + +### `ClientTimestamp` + +```rust +pub struct ClientTimestamp { + pub mono: Instant, + pub wall: SystemTime, +} +``` + +Both captured together, as close to the syscall as possible. + +### `PacketMeta` + +```rust +pub struct PacketMeta { + pub traffic_class: Option, + pub dscp: Option, + pub ecn: Option, + pub kernel_rx_timestamp: Option, +} +``` + +All `None` in MVP. Populated by `ancillary` feature. + +### `ReceivedStatsSample` + +```rust +pub struct ReceivedStatsSample { + pub count: Option, + pub window: Option, +} +``` + +### `ServerTiming` + +```rust +pub struct ServerTiming { + pub receive_wall: Option, // ns since epoch + pub receive_mono: Option, // ns duration + pub send_wall: Option, + pub send_mono: Option, + pub midpoint_wall: Option, + pub midpoint_mono: Option, +} +``` + +### `OneWayDelaySample` + +```rust +pub struct OneWayDelaySample { + pub send_delay: Duration, // server_best_receive_wall - client_send_wall + pub receive_delay: Duration, // client_receive_wall - server_best_send_wall +} +``` + +Only computable with wall clock timestamps and assumes synchronized clocks. + +### `LossKind` + +```rust +pub enum LossKind { + Unknown, + Upstream, + Downstream, +} +``` + +Classified using received window when available. + +### `ClientWarning` + +```rust +pub enum ClientWarning { + ServerProcessingExceedsRawRtt { + seq: u64, + raw: Duration, + server_processing: Duration, + }, + SubscriberDroppedEvents { + subscriber_id: SubscriberId, + count: u64, + }, + ServerRestrictedParams { + original: Params, + negotiated: Params, + }, +} +``` + +### `SessionOutcome` + +```rust +pub struct SessionOutcome { + pub end_reason: SessionEndReason, + pub packets_sent: u64, + pub replies_received: u64, + pub duplicates: u64, + pub late: u64, + pub malformed: u64, +} +``` + +### `SessionEndReason` + +```rust +pub enum SessionEndReason { + TestComplete, + Cancelled, + ServerClosed, + Error(ClientErrorKind), +} +``` + +### Sequence Number Modeling + +```rust +pub struct ProbeSeq { + pub logical: u64, + pub wire: u32, +} +``` + +Wire sequence is the 32-bit value on the wire. Logical sequence is a monotonic u64 that survives wire wraps in very long continuous sessions. For practical purposes in protocol v1, `wire = (logical as u32)`. + +--- + +## 8. Socket and Networking Plan + +### socket2 Usage + +Use `socket2::Socket` for creation and configuration: +1. Create `Socket::new(domain, Type::DGRAM, Some(Protocol::UDP))` +2. Domain: `Domain::IPV4` or `Domain::IPV6` based on resolved address +3. Bind: `socket.bind(&local_addr.into())` +4. Configure options (TOS/TC, TTL, etc.) +5. Connect: `socket.connect(&server_addr.into())` +6. Convert: `std::net::UdpSocket::from(socket)` + +### Bind/Connect Flow + +1. Resolve server address (may involve DNS) +2. Determine IP version from resolved address +3. Create socket2 socket for that IP version +4. Bind to configured local address or `0.0.0.0:0` / `[::]:0` +5. Set socket options +6. Connect to server address +7. Convert to std `UdpSocket` + +### IPv4/IPv6 Handling + +- If user specifies `-4` or `-6`, use only that version +- If user specifies an IP literal, use matching version +- If user specifies a hostname, resolve and prefer based on what succeeds (implementation-defined preference) +- Do NOT create dual-stack sockets; create one socket for one IP version per session + +### Connected UDP Semantics + +The socket is connected via `connect()`. This means: +- `send()` instead of `send_to()` — no per-packet address overhead +- `recv()` instead of `recv_from()` — kernel filters by connected address +- ICMP errors may be delivered (port unreachable on localhost) + +### DSCP / Traffic Class Setting + +- IPv4: `socket.set_tos(dscp_byte)` via socket2 +- IPv6: `socket2` does not directly expose `IPV6_TCLASS`; may need raw `setsockopt` via `libc` or `socket2`'s `set_traffic_class_v6` (check availability) +- Applied only before echo phase (after open, before first send_probe) +- Reset to 0 before close packet (per Finding C) +- Set from the negotiated DSCP value, which is the full TOS/TC byte (0–255) + +### TTL / Hop Limit + +Optional. If configured: +- IPv4: `socket.set_ttl(ttl)` via socket2 +- IPv6: `socket.set_unicast_hops_v6(hops)` via socket2 + +### Socket Cloning for Managed Mode + +`socket.try_clone()` returns a new handle to the same underlying socket. Sender and receiver threads each get a clone. This is safe for connected UDP: one thread sends, one thread receives. + +### Nonblocking / Read-Timeout Behavior + +- Manual mode: caller sets read timeout via `Client` API or uses nonblocking mode +- Managed mode receiver: uses blocking recv with a moderate timeout (e.g., 100ms) to periodically check cancellation token +- Managed mode sender: uses `thread::sleep` / `thread::park_timeout` for timing + +### Future Ancillary Feature + +Feature-gated (`ancillary`): +- Use `recvmsg` syscall to receive control messages +- Extract `IP_TOS` / `IPV6_TCLASS` from ancillary data → populate `PacketMeta` +- Extract `SO_TIMESTAMPNS` / `SCM_TIMESTAMPNS` → populate kernel RX timestamp +- Platform-specific: Linux via `rustix` or `libc`, macOS may differ +- NOT MVP — defer entirely + +### MVP vs Future + +| Feature | MVP | Future | +|---------|-----|--------| +| socket2 create/bind/connect | Yes | - | +| IPv4 TOS setting | Yes | - | +| IPv6 Traffic Class setting | Yes (best effort) | - | +| Connected UDP send/recv | Yes | - | +| Socket cloning | Yes | - | +| Read timeout | Yes | - | +| Incoming TC/DSCP/ECN | No (all None) | ancillary feature | +| Kernel RX timestamp | No (None) | ancillary feature | +| DF bit | No | Maybe | + +--- + +## 9. Statistics Plan + +### Architecture + +`StatsCollector` consumes `ClientEvent` values and maintains running statistics. It does not own threads or channels — it is a passive processor. + +```rust +pub struct StatsCollector { + config: StatsConfig, + cumulative: CumulativeStats, + rolling_count: Option, + rolling_time: Option, +} + +impl StatsCollector { + pub fn new(config: StatsConfig) -> Self; + pub fn process(&mut self, event: &ClientEvent); + pub fn cumulative(&self) -> &CumulativeStats; + pub fn rolling_count(&self) -> Option<&RollingCountWindow>; + pub fn rolling_time(&self) -> Option<&RollingTimeWindow>; + pub fn summary(&self) -> FiniteSummary; +} +``` + +### Cumulative Stats + +Maintained incrementally as events arrive: +- `packets_sent: u64` +- `replies_received: u64` +- `duplicates: u64` +- `late: u64` +- `lost: u64` +- `rtt_stats: RunningStats` (min/max/mean/variance online) +- `raw_rtt_stats: RunningStats` +- `adjusted_rtt_stats: RunningStats` (only from samples where adjusted is Some) +- `ipdv_stats: RunningStats` +- `server_processing_stats: RunningStats` +- `send_delay_stats: Option` +- `receive_delay_stats: Option` +- `upstream_lost: u64` +- `downstream_lost: u64` + +### Rolling Count Window + +A fixed-size ring buffer of the last N samples. Recomputes stats over the window on demand. + +### Rolling Time Window + +A time-bounded deque of samples. Evicts samples older than the window duration. Recomputes stats over remaining samples. + +### Finite Summary + +Built from cumulative stats at session end: +- All cumulative fields +- Median RTT (requires storing all RTT values — only for finite mode) +- Standard deviation / variance (from online algorithm) + +### RTT Stats Breakdown + +- **Effective RTT stats:** uses `rtt.effective()` for each sample +- **Raw RTT stats:** uses `rtt.raw` +- **Adjusted RTT stats:** uses `rtt.adjusted` when Some; skip when None + +### IPDV Stats + +IPDV computed as `rtt_effective[N] - rtt_effective[N-1]` for successive received replies. Only computable when both N and N-1 have replies. + +### Server Processing Stats + +From `rtt.server_processing` when Some. + +### Loss/Duplicate/Late Stats + +Simple counters. Loss percentage = `lost / (sent)`. Direction breakdown using `LossKind`. + +### Median / Stddev / Variance Strategy + +- **Online mean/variance:** Welford's algorithm (standard approach, no attribution needed) +- **Median:** Requires all values. For finite mode: store all effective RTT values in a Vec, sort at summary time. For continuous/rolling mode: median is either not available or approximate. +- **Stddev:** `sqrt(variance)` from online computation + +### Handling Edge Cases + +- **Missing adjusted RTT:** Skip in adjusted stats; effective falls back to raw, so effective stats always have data. +- **Server processing > raw RTT:** Warning already emitted by client. Stats treats adjusted as None for that sample. Effective = raw. +- **Loss direction unknown:** Counted in total loss and unknown-direction loss. +- **Duplicate replies:** Counted but not included in RTT stats (per spec: duplicates MUST NOT update original data). +- **Late replies:** Counted. `LateReply` event may include RTT if it can be computed — if so, include in late-RTT stats but NOT in primary RTT stats. +- **Events missing due to subscriber drops:** Stats will be inaccurate. `StatsCollector` should track a `missed_events` counter if the subscriber reports drops. Stats output should note that results are approximate. + +--- + +## 10. CLI Plan + +### Command Structure + +``` +irtt-cli [OPTIONS] +irtt-cli --no-test +irtt-cli --continuous +``` + +Single binary, single command (no subcommands in MVP). Options control mode and output. + +### Run Modes + +- **Finite test (default):** `-d 60s -i 1s` — run for duration, print summary +- **No-test / check:** `-n` or `--no-test` — open, negotiate, close immediately +- **Continuous:** `--continuous` or `-C` — run until Ctrl-C +- **Count:** `-c 100` — send exactly N probes + +### Output Formats + +- **human** (`--format=human`): Human-readable per-line output + summary +- **machine** (`--format=machine`): Tab-delimited, all fields, stable format +- **simple** (`--format=simple`): `timestamp\tseq\tresult` +- **rtt-us** (`--format=rtt-us`): One value per line: RTT in microseconds or "loss" +- **jsonl** (`--format=jsonl`, requires `json` feature): One JSON object per line + +Default: `human` for finite tests, `machine` for continuous + +### Machine Format Examples + +``` +reply 1714320000.123456 1.2.3.4:2112 1 42 8341 8420 79 - - - ok +loss 1714320000.323456 1.2.3.4:2112 1 43 unknown +duplicate 1714320000.423456 1.2.3.4:2112 1 42 +late 1714320000.523456 1.2.3.4:2112 1 40 8500 +``` + +### Simple Format Examples + +``` +1714320000.123456 42 8341 +1714320000.323456 43 loss +``` + +### RTT-US Format Examples + +``` +8341 +loss +8120 +``` + +### Awk-Friendly Usage + +```bash +# Extract RTT values only (rtt-us format) +irtt-cli --format=rtt-us server | grep -v loss + +# Average RTT from machine format +irtt-cli --format=machine -d 10s server | awk -F'\t' '/^reply/ { sum+=$6; n++ } END { print sum/n }' + +# Filter high-latency replies +irtt-cli --format=simple server | awk -F'\t' '$3 > 10000 { print }' +``` + +### Delimiter Support + +`--delimiter=`: tab (default), comma, semicolon, pipe, space + +### Timestamp Support + +`--timestamp-format=`: `unix` (default: `1714320000.123456`), `iso8601`, `none` + +### Header Support + +`--header`: Print column header as first line (off by default) + +### Stats Integration + +Default build (`stats` feature): +- Finite test: print per-probe lines during test, print summary at end +- Continuous: print per-probe lines, optional periodic summary lines + +Minimal build (no `stats` feature): +- Per-probe lines only +- No summary +- Smaller binary + +### Ctrl-C Handling + +Use `ctrlc` crate: +1. Register handler that sets a shared `AtomicBool` +2. On Ctrl-C: call `session.stop()` +3. Wait for `session.join()` +4. Print partial summary if stats available +5. Exit with appropriate code + +### Exit Codes + +- 0: Success +- 1: Error (open timeout, server rejection, etc.) +- 2: Partial results (cancelled mid-test) + +--- + +## 11. Feature Flags + +### `irtt-proto` + +| Feature | Default | MVP | Description | +|---------|---------|-----|-------------| +| `serde` | No | Deferred | Serialize/Deserialize on protocol types | + +### `irtt-client` + +| Feature | Default | MVP | Description | +|---------|---------|-----|-------------| +| `serde` | No | Deferred | Serialize/Deserialize on event types | +| `tracing` | No | Deferred | Structured logging | +| `ancillary` | No | Deferred | Platform control-message APIs | + +### `irtt-stats` + +| Feature | Default | MVP | Description | +|---------|---------|-----|-------------| +| `serde` | No | Deferred | Serialize/Deserialize on stats types | + +### `irtt-cli` + +| Feature | Default | MVP | Description | +|---------|---------|-----|-------------| +| `stats` | Yes | Milestone 7–8 | Include irtt-stats | +| `json` | No | Deferred | JSONL output | +| `tracing` | No | Deferred | Pass-through to irtt-client | +| `ancillary` | No | Deferred | Pass-through to irtt-client | +| `interop` | No | MVP | Enable interop test compilation | + +### MVP Features + +Milestone 1–5: No optional features needed. Core functionality only. +Milestone 6–8: `stats` feature implemented and default-enabled. +Post-MVP: `serde`, `json`, `tracing`, `ancillary`, `interop`. + +`interop` is special: it gates `#[cfg(feature = "interop")]` test modules but adds no runtime code. + +--- + +## 12. Error Model + +### `irtt-proto::ProtoError` + +```rust +#[derive(Debug, thiserror::Error)] +pub enum ProtoError { + #[error("invalid magic bytes")] + InvalidMagic, + #[error("reserved flag bits set: {0:#04x}")] + ReservedFlagBits(u8), + #[error("buffer too short: need {need}, have {have}")] + BufferTooShort { need: usize, have: usize }, + #[error("varint overflow")] + VarintOverflow, + #[error("varint truncated")] + VarintTruncated, + #[error("unknown received stats value: {0}")] + UnknownReceivedStats(i64), + #[error("unknown stamp_at value: {0}")] + UnknownStampAt(i64), + #[error("unknown clock value: {0}")] + UnknownClock(i64), + #[error("invalid HMAC")] + InvalidHmac, + #[error("midpoint and receive/send timestamps mutually exclusive")] + TimestampConflict, + #[error("server fill too long: {0} bytes")] + ServerFillTooLong(usize), + #[error("invalid parameter tag: {0}")] + InvalidParamTag(u64), +} +``` + +### `irtt-client::ClientError` + +```rust +#[derive(Debug, thiserror::Error)] +pub enum ClientError { + // Connection errors + #[error("DNS resolution failed: {0}")] + ResolutionFailed(String), + #[error("socket creation failed: {0}")] + SocketCreation(#[source] io::Error), + #[error("socket bind failed: {0}")] + SocketBind(#[source] io::Error), + #[error("socket connect failed: {0}")] + SocketConnect(#[source] io::Error), + #[error("DSCP/TOS not supported: {0}")] + DscpNotSupported(#[source] io::Error), + + // Open phase errors + #[error("open timeout: no reply after {attempts} attempts")] + OpenTimeout { attempts: usize }, + #[error("server rejected session")] + ServerRejected, + #[error("protocol version mismatch: expected {expected}, got {got}")] + ProtocolVersionMismatch { expected: i64, got: i64 }, + #[error("server restricted parameters")] + ServerRestriction { original: Params, negotiated: Params }, + + // Protocol errors + #[error("protocol error: {0}")] + Protocol(#[from] ProtoError), + + // Runtime errors + #[error("send failed: {0}")] + SendFailed(#[source] io::Error), + #[error("receive failed: {0}")] + RecvFailed(#[source] io::Error), + #[error("malformed packet from active session")] + MalformedSessionPacket, + #[error("unexpected sequence number: {0}")] + UnexpectedSequence(u32), + #[error("short reply: expected {expected}, got {got}")] + ShortReply { expected: usize, got: usize }, + #[error("stamp_at mismatch in reply")] + StampAtMismatch, + #[error("clock mismatch in reply")] + ClockMismatch, + + // Thread errors (managed mode) + #[error("worker thread panicked")] + ThreadPanic, + #[error("internal channel disconnected")] + ChannelDisconnected, + + // Partial outcome + #[error("session ended with error: {0}")] + SessionError(Box), +} +``` + +### `ClientErrorKind` + +A clonable, non-source-chaining variant of error info for embedding in `ClientEvent::FatalError`: + +```rust +#[derive(Debug, Clone)] +pub enum ClientErrorKind { + OpenTimeout, + ServerRejected, + ProtocolVersionMismatch { expected: i64, got: i64 }, + SendFailed(String), + RecvFailed(String), + MalformedSessionPacket, + ThreadPanic, + ChannelDisconnected, +} +``` + +--- + +## 13. Testing Plan + +### Pure Protocol Unit Tests (`irtt-proto`) + +- Varint encode/decode: round-trip for 0, 1, -1, 127, 128, -128, i64::MAX, i64::MIN, all verified values from black-box report +- Uvarint encode/decode: round-trip for 0, 1, 127, 128, 16383, u64::MAX +- Parameter encoding: encode known params, compare bytes to black-box captures +- Parameter decoding: decode known bytes, compare to expected params +- Parameter round-trip: encode → decode → compare +- Flags construction and parsing +- Layout calculation: verify minimum sizes for all 6 verified configurations (16, 28, 32, 44, 60, 76 bytes) +- Open request encoding: verify no token field, correct magic/flags +- Close request encoding: verify 12 bytes (no HMAC), 28 bytes (with HMAC) +- HMAC calculation: verify test vector 18.1 (echo request) +- HMAC verification: verify test vector 18.2 (echo reply) +- Field ordering: encode echo request with various field configs, verify byte offsets +- Zigzag edge cases: 0, -1, 1, i64::MAX, i64::MIN + +### Packet Vector Tests + +From `docs/test-vectors.md` (12 vectors) and protocol spec Section 18 (2 vectors): + +- Vector 1: Open request (no HMAC, 24 bytes) — verify magic, flags, param encoding +- Vector 2: Open reply (no HMAC, 32 bytes) — verify token extraction, param decoding +- Vector 3: Echo request (no HMAC, 60 bytes) — verify field layout, zeroed placeholders +- Vector 4: Echo reply (no HMAC, 60 bytes) — verify all field extraction (seq, count, window, timestamps) +- Vector 5: Close request (no HMAC, 12 bytes) — verify minimal close format +- Vector 6: HMAC open request (40 bytes) — verify HMAC computation +- Vector 7: HMAC echo request (76 bytes) — verify HMAC with full header +- Vector 8: HMAC close request (28 bytes) — verify HMAC close format +- Vector 9: No-test open+close request (25 bytes) — verify Open|Close flags, param encoding +- Vector 10: No-test open+close reply (33 bytes) — verify zero token, Open|Reply|Close flags +- Vector 11: Minimal echo packet (16 bytes) — verify absolute minimum size +- Vector 12: Midpoint timestamp echo reply (44 bytes) — verify midpoint field layout +- Spec 18.1: Echo request with HMAC (92 bytes, pattern fill) — verify HMAC, full field layout +- Spec 18.2: Echo reply with HMAC (92 bytes, pattern fill) — verify HMAC, all field values + +### Client State Tests (`irtt-client`) + +- Session phase transitions: Opening → Active → Draining → Closing → Completed +- Pending probe map: insert, match reply, detect duplicate, detect late, eviction at capacity +- Loss detection: timeout-based loss, multiple-of-interval loss +- Sequence numbering: wire/logical mapping, wrap behavior +- Negotiation policy: accept restrictions, reject restrictions, require-within bounds +- RTT computation: raw, server_processing, adjusted, effective, warning on processing > raw +- Loss direction classification from received window +- Run mode: count completion, duration completion, continuous no-stop + +### Socket Tests + +- (Minimal: socket creation is hard to unit test without a real network stack) +- Verify DSCP setter works on available platform (integration-level) +- Verify socket clone produces working handle + +### Managed Runner Tests + +- Start/stop lifecycle +- Event delivery to subscriber +- Multiple subscribers receive all events +- Subscriber overflow policies (DropNewest, DropOldest) +- Cancellation token stops all threads +- Thread panic is detected and reported + +### Stats Tests (`irtt-stats`) + +- Process sequence of EchoReply events, verify cumulative min/max/mean +- Process with losses, verify loss counts and percentages +- Process with duplicates, verify duplicate counting (not included in RTT stats) +- Rolling count window: verify correct window behavior +- Rolling time window: verify eviction +- IPDV computation from successive replies +- Edge case: server_processing > raw (adjusted=None, effective=raw) +- Edge case: all packets lost (no RTT stats) +- Edge case: single packet (no IPDV) +- Median computation for finite summary + +### CLI Formatter Tests + +- Machine format: verify tab-delimited output for EchoReply, EchoLoss, DuplicateReply, LateReply +- Simple format: verify timestamp + seq + result +- RTT-us format: verify single value or "loss" +- Delimiter customization +- Timestamp formatting +- Header line generation +- Edge cases: missing fields (no server timestamps → "-") + +### Interop Tests (feature = "interop") + +Require `irtt` server binary in PATH. Gated behind `--features interop`. + +- Basic connectivity: open, send probes, receive replies, close +- No-test mode: open+close with zero token +- Parameter negotiation (strict and loose) +- HMAC authentication (correct key) +- HMAC mismatch (expect timeout) +- DSCP setting +- All timestamp modes (none, send, receive, both, midpoint) +- All clock modes (wall, mono, both) +- All received stats modes (none, count, window, both) +- Large packet (1472 bytes) +- Server fill +- IPv6 (if available) +- Finite duration count verification (ceil(d/i)) +- Continuous mode start/stop + +### Black-Box Regression Tests + +- Run interop tests and compare packet counts, RTT ranges, error behavior to known-good baseline +- Verify close packet is sent at end of session + +### Tests Intentionally Deferred + +- Ancillary metadata tests (requires ancillary feature) +- ECN-specific tests (requires server --ecn) +- Server close during test (hard to trigger) +- Kernel RX timestamp tests (requires ancillary feature) +- Tokio/async integration tests (not planned for MVP) +- Performance/benchmark tests (post-MVP) + +### Test Execution Requirements + +- `cargo test`: Must pass without `irtt` in PATH. Pure unit tests only. +- `cargo test --workspace --features interop`: Requires `irtt` in PATH. CI must provide it. +- `tshark`: Diagnostic/manual only. Never a CI or test dependency. + +--- + +## 14. Milestone Plan + +### Milestone 0 — Workspace Skeleton + +**Goal:** Create the Cargo workspace with all four crates, license files, and documentation. + +**Files/modules:** +- `Cargo.toml` (workspace) +- `crates/irtt-proto/Cargo.toml`, `crates/irtt-proto/src/lib.rs` +- `crates/irtt-client/Cargo.toml`, `crates/irtt-client/src/lib.rs` +- `crates/irtt-stats/Cargo.toml`, `crates/irtt-stats/src/lib.rs` +- `crates/irtt-cli/Cargo.toml`, `crates/irtt-cli/src/main.rs` +- `LICENSE-APACHE`, `LICENSE-MIT` + +**Tasks:** +1. Create workspace `Cargo.toml` with members +2. Create each crate with minimal `Cargo.toml` and empty lib/main +3. Add license files +4. Add `.gitignore` +5. Verify `cargo check --workspace` passes +6. Verify `cargo test --workspace` passes (trivially) + +**Tests:** `cargo check --workspace` + +**Success criteria:** Workspace compiles. All four crates exist. No upstream source present. + +**Risks:** None. + +### Milestone 1 — `irtt-proto` + +**Goal:** Complete wire protocol implementation with all packet types and passing test vectors. + +**Files/modules:** +- `crates/irtt-proto/src/lib.rs` +- `crates/irtt-proto/src/varint.rs` +- `crates/irtt-proto/src/params.rs` +- `crates/irtt-proto/src/header.rs` +- `crates/irtt-proto/src/layout.rs` +- `crates/irtt-proto/src/hmac.rs` +- `crates/irtt-proto/src/error.rs` +- `crates/irtt-proto/src/open.rs` +- `crates/irtt-proto/src/echo.rs` +- `crates/irtt-proto/src/close.rs` + +**Tasks:** +1. Implement varint/zigzag encode/decode +2. Implement parameter model and serialization +3. Implement flags and magic constants +4. Implement packet layout calculation +5. Implement open request encoding +6. Implement open reply decoding +7. Implement echo request encoding +8. Implement echo reply decoding +9. Implement close request encoding +10. Implement HMAC-MD5 calculation and verification +11. Write all unit tests +12. Verify test vectors 18.1 and 18.2 + +**Tests:** +- All varint round-trips +- All parameter encode/decode tests +- Layout calculations for all 6 verified configurations +- All 14 packet test vectors (12 from `docs/test-vectors.md` + 2 from spec Section 18) +- HMAC verification for vectors 6, 7, 8, spec 18.1, spec 18.2 +- Edge cases (empty params, unknown tags, max values) + +**Success criteria:** All proto tests pass. All 14 test vectors verified. `cargo test -p irtt-proto` green. + +**Risks:** Varint encoding mismatch (mitigated by verified test values). + +### Milestone 2 — Manual Client No-Test / Open + +**Goal:** `Client` can connect to an irtt server, perform open handshake, and run no-test mode. + +**Files/modules:** +- `crates/irtt-client/src/lib.rs` +- `crates/irtt-client/src/socket.rs` +- `crates/irtt-client/src/config.rs` +- `crates/irtt-client/src/session.rs` +- `crates/irtt-client/src/client.rs` +- `crates/irtt-client/src/event.rs` +- `crates/irtt-client/src/error.rs` +- `crates/irtt-client/src/timing.rs` + +**Tasks:** +1. Implement `ClientSocket` with socket2 +2. Implement `ClientConfig` and `SocketConfig` +3. Implement DNS resolution +4. Implement `Client::connect` +5. Implement open request sending with retransmission +6. Implement open reply parsing and validation +7. Implement negotiation policy +8. Implement no-test mode (Open+Close flags) +9. Implement `Client::close` +10. Define `ClientEvent` enum (full definition, even if only SessionStarted/SessionEnded used here) +11. Define error types + +**Tests:** +- Unit: config validation, negotiation policy logic +- Interop (gated): no-test mode against real server +- Interop (gated): basic open/close cycle + +**Success criteria:** `Client::connect` + `Client::open` + no-test mode works against irtt server. Interop test passes. + +**Risks:** Socket option availability on target platforms. IPv6 TOS/TC setting may need platform-specific handling. + +### Milestone 3 — Manual Finite Probe Events + +**Goal:** `Client` can send probes, receive replies, and emit events for a finite test. + +**Files/modules:** +- `crates/irtt-client/src/probe.rs` (pending probe map) +- `crates/irtt-client/src/validate.rs` +- Updates to `client.rs`, `session.rs`, `event.rs` + +**Tasks:** +1. Implement `Client::send_probe` +2. Implement `Client::recv_once` +3. Implement `Client::recv_available` +4. Implement `Client::poll_timeouts` +5. Implement pending probe map (bounded) +6. Implement reply matching and validation +7. Implement RTT sample computation +8. Implement duplicate detection +9. Implement late detection +10. Implement loss timeout detection +11. Implement `RunMode::Duration` and `RunMode::Count` completion logic +12. Implement `OneWayDelaySample` computation +13. Implement `ReceivedStatsSample` extraction +14. Implement `ServerTiming` extraction +15. Implement `ClientWarning::ServerProcessingExceedsRawRtt` + +**Tests:** +- Unit: RTT computation (raw, adjusted, effective) +- Unit: pending probe map operations +- Unit: duplicate/late detection +- Unit: loss timeout detection +- Unit: received window → loss direction classification +- Interop (gated): finite 5s test, verify packet counts, RTT values present +- Interop (gated): verify `ceil(d/i)` packet count formula + +**Success criteria:** Manual client can run a complete finite test and produce correct events. Interop tests pass. + +**Risks:** Timing accuracy. Potential edge cases in recv_once blocking behavior. + +### Milestone 4 — Minimal CLI Stream Mode + +**Goal:** Working CLI binary that can run a test and print results in machine/simple/rtt-us format. + +**Files/modules:** +- `crates/irtt-cli/src/main.rs` +- `crates/irtt-cli/src/args.rs` +- `crates/irtt-cli/src/format.rs` +- `crates/irtt-cli/src/run.rs` +- `crates/irtt-cli/src/signal.rs` + +**Tasks:** +1. Implement CLI argument parsing with clap +2. Implement machine output formatter +3. Implement simple output formatter +4. Implement rtt-us output formatter +5. Implement basic run loop (using manual Client initially, or managed if M5 is done) +6. Implement Ctrl-C handling +7. Implement delimiter support +8. Implement timestamp formatting +9. Implement header option + +**Tests:** +- Unit: formatter tests for each format +- Unit: delimiter substitution +- Unit: timestamp formatting +- Integration: run CLI against irtt server, verify output format +- Verify awk-friendly examples from design doc work + +**Success criteria:** `irtt-cli ` runs a test and prints machine-format output. Ctrl-C works. + +**Risks:** None significant. Straightforward formatting work. + +### Milestone 5 — Managed Runner + +**Goal:** `ManagedClient` with coordinator/sender/receiver threads and event fanout. + +**Files/modules:** +- `crates/irtt-client/src/managed/mod.rs` +- `crates/irtt-client/src/managed/coordinator.rs` +- `crates/irtt-client/src/managed/sender.rs` +- `crates/irtt-client/src/managed/receiver.rs` +- `crates/irtt-client/src/managed/hub.rs` +- `crates/irtt-client/src/managed/cancellation.rs` + +**Tasks:** +1. Implement `CancellationToken` +2. Implement `EventHub` with subscriber management +3. Implement `SubscriberConfig` and overflow policies +4. Implement sender/pacer thread with absolute scheduling +5. Implement receiver thread +6. Implement coordinator thread (correlate, compute RTT, emit events) +7. Implement `ManagedClient::start` (open on calling thread, then spawn) +8. Implement `ManagedClientSession::subscribe` +9. Implement `ManagedClientSession::stop` and `join` +10. Implement continuous mode +11. Implement thread panic detection +12. Implement graceful shutdown (close packet) + +**Tests:** +- Unit: EventHub fanout to multiple subscribers +- Unit: SubscriberOverflow policies +- Unit: CancellationToken +- Integration: ManagedClient finite test, verify events arrive at subscriber +- Integration: ManagedClient continuous mode, stop, verify clean shutdown +- Interop (gated): managed client against real server + +**Success criteria:** ManagedClient runs tests, delivers events to subscribers, shuts down cleanly. + +**Risks:** Thread coordination complexity. Potential deadlocks if channels are misused. Careful testing required. + +### Milestone 6 — `irtt-stats` + +**Goal:** Statistics crate that consumes events and produces cumulative/rolling/summary stats. + +**Files/modules:** +- `crates/irtt-stats/src/lib.rs` +- `crates/irtt-stats/src/collector.rs` +- `crates/irtt-stats/src/running.rs` +- `crates/irtt-stats/src/rolling.rs` +- `crates/irtt-stats/src/summary.rs` +- `crates/irtt-stats/src/median.rs` + +**Tasks:** +1. Implement `RunningStats` (online mean/variance) +2. Implement `StatsCollector` event processing +3. Implement cumulative stats +4. Implement rolling count window +5. Implement rolling time window +6. Implement IPDV computation +7. Implement loss/duplicate/late counting +8. Implement finite summary with median +9. Handle edge cases (missing adjusted, processing > raw, all lost) + +**Tests:** +- Unit: RunningStats with known sequences +- Unit: process events, verify min/max/mean/variance +- Unit: rolling window behavior +- Unit: IPDV from successive RTTs +- Unit: median computation +- Unit: edge cases (empty, single, all lost) + +**Success criteria:** StatsCollector produces correct statistics from event streams. + +**Risks:** Floating-point precision. Median storage for long tests (acceptable for finite mode). + +### Milestone 7 — Full/Default CLI Stats Integration + +**Goal:** Default CLI build includes stats. Finite tests show summary. Minimal build still works. + +**Files/modules:** +- Updates to `crates/irtt-cli/src/run.rs` +- `crates/irtt-cli/src/format.rs` (summary formatting) +- `crates/irtt-cli/Cargo.toml` (feature flags) + +**Tasks:** +1. Wire StatsCollector into CLI run loop as event subscriber +2. Implement human summary formatting +3. Implement stats display for finite tests +4. Optional periodic stats in continuous mode +5. Verify `--no-default-features` build works without stats +6. Implement human output format (with summary) + +**Tests:** +- Integration: finite test with summary output +- Integration: minimal build compiles and runs without stats +- Unit: human summary formatting + +**Success criteria:** `cargo build -p irtt-cli` includes stats. `cargo build -p irtt-cli --no-default-features` works without. + +**Risks:** None significant. + +### Milestone 8 — Compatibility Features + +**Goal:** Full protocol compatibility: HMAC, all timestamp modes, all received stats modes, server fill, DSCP. + +**Files/modules:** +- Updates across irtt-client and irtt-cli + +**Tasks:** +1. HMAC interop testing (correct key, wrong key, no key) +2. All StampAt modes (none, send, receive, both, midpoint) +3. All Clock modes (wall, mono, both) +4. All ReceivedStats modes (none, count, window, both) +5. Server fill +6. DSCP / Traffic Class setting +7. TTL / hop limit (if desired) +8. Large packet sizes +9. Comprehensive interop test suite + +**Tests:** +- Interop: all timestamp × clock × stats combinations +- Interop: HMAC authentication +- Interop: DSCP capture verification +- Interop: server fill +- Interop: large packets + +**Success criteria:** Full compatibility with irtt 0.9.1 server across all parameter combinations. + +**Risks:** Platform-specific socket option issues (IPv6 TC, DF bit). + +### Milestone 9 — Advanced Networking + +**Goal:** Ancillary receive metadata for ECN/DSCP/kernel RX timestamps. + +**Files/modules:** +- `crates/irtt-client/src/ancillary.rs` (feature-gated) +- Platform-specific code + +**Tasks:** +1. Implement recvmsg wrapper (Linux) +2. Extract IP_TOS / IPV6_TCLASS from ancillary data +3. Extract kernel RX timestamp +4. Populate PacketMeta +5. macOS support (if feasible) +6. Test with `irtt server --ecn` + +**Tests:** +- Integration: verify ancillary data extraction on Linux +- Interop: verify ECN behavior with server --ecn +- Unit: ancillary data parsing + +**Success criteria:** PacketMeta populated with real values on supported platforms. + +**Risks:** High platform variability. macOS ancillary API differences. ECN behavior not fully verified. + +--- + +## 15. Open Questions + +### Q1: Server Close During Active Test (Section 19.3) + +**Why it matters:** If the server can set Close flag on echo replies mid-test, the client must detect and handle this gracefully. + +**Blocks implementation:** No. Defensive handling is straightforward. + +**Suggested behavior:** Check for Close flag in every echo reply. If set, process the reply normally, emit SessionEnded with reason ServerClosed. This is safe regardless of whether the behavior actually occurs. + +**Black-box test suggestion:** Run a very long test exceeding the server's max duration. If the server closes the session, observe the packet. + +### Q2: Send Timestamp Capture Timing (Section 19.6) + +**Why it matters:** Affects RTT accuracy. + +**Blocks implementation:** No. This is an implementation choice. + +**Recommended behavior:** Capture `Instant::now()` + `SystemTime::now()` immediately after `send()` returns. This is what the spec recommends and is the simplest correct approach. + +### Q3: RTT When Server Processing > Raw RTT (Section 19.12) + +**Why it matters:** Negative adjusted RTT is nonsensical. + +**Blocks implementation:** No. The Rust design already specifies the correct handling. + +**Recommended behavior:** Set `adjusted = None`, emit `ClientWarning::ServerProcessingExceedsRawRtt`. `effective()` falls back to raw. This is already in the design. + +### Q4: Upstream Server `--ecn` + +**Why it matters:** Required for full ECN compatibility testing. + +**Blocks implementation:** No. ECN is a deferred ancillary feature (Milestone 9). + +**Suggested black-box test:** Run `irtt server --ecn` and capture packets with tshark. Observe whether the server sets ECN bits in IP header. Test with `--dscp` values that include ECN bits. + +### Q5: Continuous Session Duration Semantics + +**Why it matters:** If server negotiates a finite duration for a continuous client, the client must handle session expiry. + +**Blocks implementation:** No. `ReconnectPolicy` covers this. + +**Recommended behavior:** Default `ReconnectPolicy::Never` for library. CLI continuous mode uses `ReconnectPolicy::OnServerClose { delay: 1s }`. If the server closes, reconnect after delay. + +### Q6: Exact Feature-Gating of Stats/JSON/Ancillary + +**Why it matters:** Build size and dependency management. + +**Blocks implementation:** No. Feature flags are well-defined in the design. + +**Recommended behavior:** Follow the design document exactly. `stats` is default on for irtt-cli. All others are opt-in. + +### Q7: DropOldest Subscriber Overflow with Flume + +**Why it matters:** `flume` doesn't natively support drop-oldest semantics. + +**Blocks implementation:** No, but requires a design decision. + +**Options:** +1. Implement DropOldest by draining one item from the receiver before retrying send (requires access to both ends) +2. Use a different channel type for DropOldest subscribers (e.g., a ring buffer) +3. Defer DropOldest and only implement DropNewest/Block/Disconnect initially + +**Recommended temporary behavior:** Implement DropOldest as: try_send fails → recv one from the same channel → try_send again. This requires the EventHub to hold both sender and receiver for DropOldest subscribers, which is slightly unusual but workable. + +--- + +## 16. Risk Register + +| Risk | Impact | Likelihood | Mitigation | Owner/Artifact | +|------|--------|------------|------------|----------------| +| Clean-room contamination | Critical — legal/licensing | Low | Implementation agent reads only clean artifacts. No upstream source in repo. Documented boundary in CLEANROOM_NOTES.md | Process / all crates | +| Protocol wire format mismatch | High — interop failure | Low | Test vectors verified. Black-box captures available. Interop tests in CI. | irtt-proto | +| Varint/zigzag encoding mismatch | High — open/negotiation fails | Very Low | Independently verified with multiple value samples. Standard LEB128/zigzag. | irtt-proto | +| HMAC computation mismatch | High — authentication fails | Very Low | Two test vectors verified. Four packet types verified against captures. | irtt-proto | +| Timing accuracy insufficient | Medium — RTT measurements off | Low | Use monotonic clock. Absolute scheduling. Timestamp close to syscall. TimerMode::Precise available as opt-in. | irtt-client | +| Router resource exhaustion | Medium — OOM on OpenWrt | Low | Bounded pending map. No mandatory sample retention. Feature-gated stats. Minimal build path. | irtt-client, irtt-cli | +| Event subscriber backpressure delays sender | High — timing disruption | Low | Sender thread never touches event hub. Only coordinator fans out. Coordinator is not timing-critical. | irtt-client (managed) | +| Stats/event loss from subscriber drops | Medium — inaccurate stats | Medium | Document limitation. Track drop count. Stats output marks approximate results. | irtt-stats | +| Managed runner deadlock | High — hang | Low | Careful channel sizing. CancellationToken as escape hatch. Thread panic detection. Thorough testing. | irtt-client (managed) | +| ECN/ancillary portability | Medium — feature unavailable | Medium | Feature-gated. Deferred to Milestone 9. Platform-specific code isolated. Graceful fallback to None. | irtt-client (ancillary) | +| CLI format instability | Medium — downstream breakage | Low | Machine format fields are positional and documented. New fields added at end only. | irtt-cli | +| IPv6 Traffic Class socket option | Low — IPv6 DSCP may not work | Medium | Use socket2 if available. Fall back to libc setsockopt. Test on target platforms. | irtt-client | +| Server behavior divergence across versions | Low — untested server behavior | Low | Target irtt 0.9.1 only. Black-box tests as regression suite. | all | +| flume DropOldest not native | Low — subscriber policy gap | Medium | Workaround with drain-and-retry. Or defer DropOldest initially. | irtt-client (managed) | + +--- + +## 17. Implementation Agent Instructions + +### What to implement first + +1. **Milestone 0:** Workspace skeleton. Get `cargo check --workspace` passing with empty crates. +2. **Milestone 1:** `irtt-proto`. All wire format logic. This is pure, testable, and has no external dependencies beyond crypto crates. Verify test vectors before moving on. +3. **Milestone 2:** Manual `Client` open/close/no-test. First real interop with an irtt server. +4. **Milestone 3:** Manual `Client` probe sending/receiving. Full event model. + +### What NOT to implement first + +- Do NOT implement `ManagedClient` before the manual `Client` works (milestones 2–3 come before 5). +- Do NOT implement `irtt-stats` before the client produces correct events (milestone 6 comes after 5). +- Do NOT implement ancillary networking (Milestone 9) during initial implementation. +- Do NOT implement serde/json/tracing features during initial milestones. +- Do NOT implement human output format before machine format works. +- Do NOT implement `ReconnectPolicy` logic before basic continuous mode works. + +### What files to read + +- `docs/IRTT_CLIENT_PROTOCOL_SPEC.md` — normative protocol reference +- `docs/BLACKBOX_VERIFICATION_REPORT.md` — verified protocol details and packet captures +- `docs/RUST_DESIGN.md` — architecture and product goals +- `docs/test-vectors.md` — 12 packet test vectors from real captures +- `docs/CLEANROOM_NOTES.md` — clean-room boundary (read once for awareness) +- `IMPLEMENTATION_PLAN.md` (this file) — detailed implementation guidance + +### What files NOT to read + +- Any upstream IRTT source code (Go files, Go tests, Go modules) +- Any files in `captures/` directory (these are diagnostic artifacts from the spec agent, not implementation inputs — use the verified findings in the black-box report instead) +- Any contaminated notes or transcripts +- Any GPL-licensed material + +### Clean-room boundaries to maintain + +1. Never search for, open, or read upstream IRTT source files +2. Never search online for IRTT implementation details +3. Never ask for contaminated notes or transcripts +4. Never copy or adapt upstream type names, function names, or module layout +5. Design all internal names independently based on Rust conventions +6. Use only the clean protocol spec and black-box report for protocol truth +7. When uncertain about protocol behavior, mark as open question and suggest a black-box test — do NOT guess from upstream patterns + +### How to report uncertainties + +- If a protocol detail is unclear: add a `// OPEN: ` comment and note it in test output +- If a behavior differs from spec expectation during interop testing: document the discrepancy, do not silently adapt +- If a platform-specific feature is unavailable: use `cfg` attributes, provide graceful fallback, and document + +### How to structure commits/milestones + +- One commit per logical unit of work (e.g., "implement varint encoding" or "add open request encoding") +- Tag milestone completion with a clear commit message (e.g., "milestone 1: irtt-proto complete") +- Ensure `cargo test` passes at every commit +- Ensure `cargo clippy` is clean at every commit +- Run `cargo test --features interop` at milestone boundaries (2, 3, 4, 5, 8) + +--- + +## 18. Design Brief Amendments + +### Amendment 1: `irtt-stats` dependency on `irtt-client` + +The design lists `irtt-client` as a dependency of `irtt-stats`. This is correct because `irtt-stats` needs `ClientEvent` types. However, consider whether `ClientEvent` and related types should be in a separate `irtt-client-types` crate or re-exported from `irtt-proto` to avoid pulling in socket/thread code. + +**Recommendation:** Keep as-is for now. The dependency is on the types, not the runtime. Rust's dead-code elimination handles this. If binary size becomes an issue, extract types later. + +### Amendment 2: `DropOldest` subscriber overflow + +The design specifies `DropOldest` as a subscriber overflow policy. Flume doesn't natively support this. + +**Recommendation:** Implement `DropOldest` in MVP using the drain-one-and-retry approach (EventHub holds both Sender and a cloned Receiver for DropOldest subscribers). If this proves problematic, defer DropOldest to post-MVP and document the limitation. + +### Amendment 3: `Client::open` return type + +The design shows `Client::open` returning `Result, ClientError>`. For the manual API, returning a typed result struct might be cleaner than a Vec: + +**Recommendation:** Use `Result` where `OpenOutcome` contains the `SessionStarted` event and negotiated params. The Vec pattern is better suited for `send_probe`/`recv_once` which may produce multiple events. + +### Amendment 4: `recv_once` return type + +`recv_once` currently returns `Result>, ClientError>`. This is a triple-nested result (Result → Option → Vec). Consider simplifying to `Result, ClientError>` where an empty SmallVec means "no data available." + +**Recommendation:** Use `Result, ClientError>` with empty Vec meaning "no data." This is simpler and the allocation cost is negligible per-recv. Or use a custom `RecvResult` enum: `Data(Vec)`, `WouldBlock`, `Timeout`. + +### Amendment 5: `EchoSent` event + +The design says `EchoSent` should not be emitted by default. Consider not defining it at all in MVP, to avoid dead code. + +**Recommendation:** Omit `EchoSent` entirely from MVP. Add it later if needed for debugging. + +### Amendment 6: Clock value 0 + +The spec defines Clock enum values as 1=Wall, 2=Monotonic, 3=Both. Value 0 is not defined. The implementation should treat Clock=0 as equivalent to "no timestamps" (the StampAt=None case makes Clock irrelevant). + +**Recommendation:** If StampAt is None, ignore Clock value. If StampAt is non-None and Clock is 0, treat as an error or default to Wall. Document the decision. + +--- + +## 19. Final Recommendation + +**Proceed.** + +The clean artifacts are thorough and internally consistent. The protocol spec has been verified against a real irtt 0.9.1 server. The Rust design is well-structured and addresses all major architectural concerns. The three remaining open questions (server close mid-test, timestamp capture timing, processing > raw RTT) do not block implementation — defensive handling is specified for all three. + +No major blockers exist. The recommended minor amendments (Section 18) are quality-of-life improvements, not structural issues. + +**First milestone to implement:** Milestone 0 (workspace skeleton), immediately followed by Milestone 1 (`irtt-proto`). The protocol crate is pure, heavily testable, and establishes the foundation for everything else. diff --git a/docs/IRTT_CLIENT_PROTOCOL_SPEC.md b/docs/IRTT_CLIENT_PROTOCOL_SPEC.md new file mode 100644 index 0000000..d53fb34 --- /dev/null +++ b/docs/IRTT_CLIENT_PROTOCOL_SPEC.md @@ -0,0 +1,1450 @@ +# IRTT-Compatible Client Protocol Specification + +**Version:** 1.1-verified +**Date:** 2026-04-28 +**Verified against:** irtt 0.9.1, protocol version 1, macOS Darwin 25.3.0 arm64 + +--- + +## 1. Status + +This document describes the externally observable behavior required for an +independent client implementation compatible with existing IRTT (Isochronous +Round-Trip Tester) servers. + +This document does **not** describe upstream source structure, internal +algorithms, or implementation organization. It is intended for a clean-room +implementation that has never seen the upstream source code. + +The key words "MUST", "MUST NOT", "REQUIRED", "SHOULD", "SHOULD NOT", +"RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted +as interoperability requirements, consistent with RFC 2119. + +--- + +## 2. Scope + +- **Client-only implementation.** Server implementation is out of scope unless + server behavior is needed to understand client interoperability. +- **Compatible with existing IRTT servers** running protocol version 1. +- This specification describes protocol and measurement behavior; CLI design + is not prescribed. +- Server implementation is out of scope. + +--- + +## 3. Terminology + +| Term | Definition | +|------|-----------| +| **Client** | The endpoint that initiates a session, sends test packets, and receives replies. | +| **Server** | The endpoint that accepts sessions and echoes test packets back to the client. | +| **Session** | A stateful association between a client and server, identified by a connection token. | +| **Open / Negotiation phase** | The initial handshake in which the client proposes test parameters and the server responds with (possibly restricted) parameters and a connection token. | +| **Test packet** | A UDP datagram sent by the client during the active test phase. Sometimes called an "echo request." | +| **Reply packet** | A UDP datagram sent by the server in response to a test packet. Sometimes called an "echo reply." | +| **Close / Finalization phase** | The exchange in which the client notifies the server that the session is ending. | +| **Connection token** | A 64-bit value assigned by the server during negotiation, used to identify the session in all subsequent packets. | +| **Round-trip time (RTT)** | The time between sending a test packet and receiving its reply, optionally minus server processing time. | +| **Send interval** | The fixed time period between successive test packet transmissions. | +| **Sequence number** | A 32-bit unsigned integer, starting at 0, assigned to each test packet in order of transmission. | +| **Timeout** | The duration the client waits for a server reply during the open phase before retransmitting. | +| **Packet loss** | A test packet for which no reply was received. | +| **Duplicate** | A reply received with a sequence number for which a reply was already recorded. | +| **Late packet** | A reply received with a sequence number lower than the most recently received sequence number (indicates out-of-order delivery). | +| **IPDV (Instantaneous Packet Delay Variation)** | The difference in delay between two successive successfully returned replies. Commonly called "jitter." | +| **Server processing delay** | The time the server spent between receiving a request and sending its reply, measured by server timestamps. | +| **Received count** | A running count of all packets the server has received for this session. | +| **Received window** | A 64-bit bitmap indicating the receipt status of the most recent 64 packets at the server. | +| **Wall clock** | System real-time clock (nanoseconds since Unix epoch). Subject to NTP adjustments. | +| **Monotonic clock** | A clock that only moves forward and is not subject to system time adjustments. Used for duration calculations. | +| **HMAC** | Hash-based Message Authentication Code (MD5), used for packet authentication. | +| **DSCP** | Differentiated Services Code Point, set in the IP header TOS/Traffic Class field. | +| **Drain period** | The period after the last test packet is sent during which the client waits for outstanding replies. | + +--- + +## 4. Conformance + +A conforming client MUST implement: + +1. Session lifecycle: open, test, close. +2. The binary wire format defined in this document. +3. Parameter negotiation. +4. Connection token handling. +5. Sequence number assignment and reply matching. +6. RTT measurement. +7. Packet loss detection. +8. Tolerance for UDP reordering and duplication. + +A conforming client SHOULD implement: + +1. HMAC authentication. +2. Server timestamp handling (one-way delay, IPDV). +3. Received stats (count and window) processing. +4. Configurable DSCP. + +Implementations are not required to use any specific internal architecture, +data structure, function layout, naming convention, concurrency model, or +source organization. Only the externally observable behavior described in +this document is normative. + +--- + +## 5. Transport Requirements + +### 5.1 Transport Protocol + +IRTT uses **UDP** exclusively. There is no TCP fallback. + +### 5.2 Default Port + +The default server port is **2112**. The default client local port is +ephemeral (OS-assigned, port 0). + +### 5.3 IPv4 and IPv6 + +A conforming client MUST support IPv4. IPv6 support is RECOMMENDED. When the +user does not specify an IP version preference, the client SHOULD attempt +address resolution for both IPv4 and IPv6, and use whichever resolves +successfully. If both resolve, the selection behavior is +implementation-defined. + +The client MUST use a single connected UDP socket for the entire session. The +client calls the equivalent of `connect()` on the socket, binding it to the +resolved server address before the open phase. + +### 5.4 Socket Options + +- **DSCP/TOS:** A client SHOULD support setting the DSCP value on outgoing + packets. For IPv4, this is the TOS field. For IPv6, this is the Traffic + Class field. The DSCP value is negotiated as a parameter; if the server + disallows DSCP, the negotiated value will be 0. +- **TTL/Hop Limit:** A client MAY support setting the IP TTL (IPv4) or Hop + Limit (IPv6). +- **Don't Fragment (DF):** A client MAY support setting the DF bit. +- **ECN:** ECN bit handling is OPTIONAL and experimental. + +### 5.5 Connectionless Behavior + +Although the client uses a connected UDP socket, the protocol is +connectionless at the network layer. The client MUST tolerate: + +- Packet reordering. +- Packet duplication. +- Packet loss. +- Receiving unrelated UDP traffic on the socket (which MUST be discarded + based on validation rules). + +### 5.6 MTU Considerations + +Common maximum unfragmented UDP payload sizes for a 1500-byte MTU: +- IPv4: 1472 bytes (1500 - 20 IPv4 header - 8 UDP header) +- IPv6: 1452 bytes (1500 - 40 IPv6 header - 8 UDP header) + +The client does not need to perform path MTU discovery. Packet length is a +user-configured parameter. + +--- + +## 6. Session Lifecycle + +A client session progresses through the following externally observable states: + +``` + Resolving ──► Opening ──► Active Test ──► Draining ──► Closing ──► Completed + │ │ + ▼ ▼ + Failed Completed +``` + +### 6.1 Resolving + +- **Entry:** Client is given a server address (hostname or IP) and port. +- **Behavior:** The client resolves the address to a UDP endpoint. The + default port (2112) MUST be appended if no port is specified. +- **Exit:** Successful resolution transitions to Opening. Failure transitions + to Failed. + +### 6.2 Opening / Negotiating + +- **Entry:** The client has a resolved server address and a connected UDP socket. +- **Behavior:** + 1. The client constructs an open request packet (see Section 8.2) containing + the proposed test parameters serialized in the payload. + 2. The client sends the open request and waits for a reply. + 3. If no reply is received within the current timeout, the client + retransmits the open request with the next timeout value from its + timeout list. + 4. The default timeout sequence is: 1s, 2s, 4s, 8s (total maximum wait: + 15s, maximum 4 packets). The minimum timeout value MUST be at least + 200ms. + 5. If all timeouts are exhausted with no reply, the session transitions to + Failed. + +- **Valid server responses:** + - An open reply with a non-zero connection token: session parameters are + extracted from the reply payload. The client stores the connection token. + Transitions to Active Test (or Completed if the no-test flag was set). + - An open reply with the close flag set, **when the client did NOT set + Close in its request**: the server rejected the session (e.g., protocol + version mismatch). Transition to Failed. + - An open reply with the close flag set, **when the client DID set Close + in its request** (no-test mode): the server acknowledged the open+close. + The connection token is zero. Transition to Completed. + [**Verified 2026-04-28** — see BLACKBOX_VERIFICATION_REPORT.md Finding D.] + +- **Parameter negotiation:** The server MAY restrict parameters. The client + MUST compare returned parameters against its requested parameters. A + conforming client MUST, by default, reject the session if parameters were + restricted (transition to Failed with an appropriate error). A "loose" mode + MAY be provided that accepts server restrictions with a warning. + + The client MUST reject (as an error) the following server parameter changes: + - Duration increased beyond what was requested. + - Length increased beyond what was requested. + - Interval reduced below a safety floor (see Section 19.9 for verification). + +- **Protocol version check:** The client MUST verify that the server's + protocol version matches exactly. Protocol version 1 is the current + version. Mismatch MUST cause the session to fail. + +### 6.3 Active Test + +- **Entry:** Open reply received and parameters accepted. +- **Behavior:** + 1. The client sends test packets at fixed intervals for the configured + duration. + 2. The client MUST be able to receive reply packets while continuing + to send test packets (i.e., sending and receiving occur concurrently). + 3. Test packets are numbered with sequence numbers starting at 0. + 4. The test duration is **exclusive**: the client MUST NOT send a packet at + or after the end time. The end time is calculated as start time + duration. + The next scheduled send time is computed, and if it would be at or past + the end time, sending stops. + 5. The expected number of packets for a given duration `d` and interval `i` + is: `ceil(d / i)`. [**Verified 2026-04-28** by black-box testing with + irtt 0.9.1. See BLACKBOX_VERIFICATION_REPORT.md Finding A.] + +- **Exit:** All scheduled packets have been sent. Transition to Draining. + +### 6.4 Draining Replies + +- **Entry:** All test packets have been sent. +- **Behavior:** The client waits for outstanding replies. The wait duration + is configurable: + - Default: 3 times the maximum observed RTT, falling back to 4 seconds if + no RTT has been measured. + - Alternatives: a factor of mean RTT, or a fixed duration. + - If all replies have already been received, no wait is needed. + +- **Exit:** Wait period expires, or all replies received. Transition to Closing. + +### 6.5 Closing + +- **Entry:** Test and drain are complete. +- **Behavior:** The client sends a single close packet (see Section 8.4) to + the server. The close packet includes the connection token and the close + flag. The client does NOT wait for a close acknowledgement (this is a + known limitation of protocol version 1). + +- **Exit:** Close packet sent. Transition to Completed. + +### 6.6 Completed + +- **Entry:** Session has finished normally. +- **Behavior:** Results are computed and made available. + +### 6.7 Failed + +- **Entry:** An unrecoverable error occurred. +- **Behavior:** Partial results MAY be available if the failure occurred + during or after the Active Test phase. + +--- + +## 7. Packet Format Overview + +All IRTT packets share a common header structure. The presence or absence of +optional fields is determined by context (packet type and negotiated +parameters). + +| Packet Type | Direction | When Sent | Required | Purpose | +|------------|-----------|-----------|----------|---------| +| Open Request | Client → Server | Opening phase | Yes | Propose session parameters | +| Open Reply | Server → Client | Opening phase | Yes | Return connection token and (possibly restricted) parameters | +| Echo Request (Test Packet) | Client → Server | Active test | Yes | Carry test payload with sequence number | +| Echo Reply | Server → Client | Active test | Yes | Return test payload with timestamps and stats | +| Close Request | Client → Server | Closing phase | Yes | Signal session end | +| Close Reply | Server → Client | Closing phase | No | Acknowledge close (server MAY send this) | + +--- + +## 8. Binary Wire Format + +### 8.1 General Header Structure + +All multi-byte integer fields use **little-endian** byte order. + +The packet format uses a flexible field layout. Fields are placed sequentially +in the packet buffer. A field is either present at its full capacity or absent +(zero length). The set of present fields is determined by the packet type and +negotiated parameters. + +#### 8.1.1 Fixed Header Fields (always present) + +| Offset | Size (bytes) | Field | Encoding | Description | +|--------|-------------|-------|----------|-------------| +| 0 | 3 | Magic | Raw bytes | Fixed value: `0x14 0xA7 0x5B` | +| 3 | 1 | Flags | Unsigned byte | Bit flags (see below) | + +**Total minimum header: 4 bytes.** + +#### 8.1.2 Flag Bits + +| Bit | Mask | Name | Meaning | +|-----|------|------|---------| +| 0 | 0x01 | Open | Set on open request and open reply packets | +| 1 | 0x02 | Reply | Set on all packets from server to client; clear on all packets from client to server | +| 2 | 0x04 | Close | Set on close request and close reply packets | +| 3 | 0x08 | HMAC | Set if an HMAC field is present in the packet | + +Bits 4-7 are reserved and MUST be zero. A client MUST reject packets where +reserved flag bits are set. + +#### 8.1.3 Optional Fields + +After the fixed header, optional fields appear **in the following order** when +present. A field is either present at full size or entirely absent. There are +no length-prefix or presence-indicator bytes; the receiver knows which fields +are present based on the packet type and negotiated session parameters. + +**Verification note:** The field ordering below is critical for +interoperability. [**Verified 2026-04-28** by packet capture with multiple +field combinations — see BLACKBOX_VERIFICATION_REPORT.md Section 19.11.] + +| Order | Field | Size (bytes) | Encoding | When Present | +|-------|-------|-------------|----------|-------------| +| 1 | HMAC | 16 | MD5 HMAC | When HMAC flag (0x08) is set | +| 2 | Connection Token | 8 | Little-endian uint64 | All packets except initial open request sent from client before token is known. Present in open reply and all subsequent packets. | +| 3 | Sequence Number | 4 | Little-endian uint32 | Echo request and echo reply packets | +| 4 | Received Count | 4 | Little-endian uint32 | Echo reply, when "count" stats negotiated | +| 5 | Received Window | 8 | Little-endian uint64 | Echo reply, when "window" stats negotiated | +| 6 | Receive Wall Timestamp | 8 | Little-endian int64 (ns since Unix epoch) | Echo reply, when receive timestamps negotiated with wall clock | +| 7 | Receive Mono Timestamp | 8 | Little-endian int64 (ns, duration) | Echo reply, when receive timestamps negotiated with monotonic clock | +| 8 | Midpoint Wall Timestamp | 8 | Little-endian int64 (ns since Unix epoch) | Echo reply, when midpoint timestamp negotiated with wall clock | +| 9 | Midpoint Mono Timestamp | 8 | Little-endian int64 (ns, duration) | Echo reply, when midpoint timestamp negotiated with monotonic clock | +| 10 | Send Wall Timestamp | 8 | Little-endian int64 (ns since Unix epoch) | Echo reply, when send timestamps negotiated with wall clock | +| 11 | Send Mono Timestamp | 8 | Little-endian int64 (ns, duration) | Echo reply, when send timestamps negotiated with monotonic clock | + +**Important:** Midpoint timestamps are **mutually exclusive** with +receive/send timestamps. If a midpoint timestamp is present, receive and send +timestamp fields MUST NOT be present, and vice versa. A packet containing +both midpoint and receive/send timestamps is malformed. + +When both receive and send timestamps are present, they MUST use the same +clock mode (both wall, both monotonic, or both wall+monotonic). + +#### 8.1.4 Payload + +Everything after the last present header field is **payload**. The total +packet length is the negotiated packet length. If the negotiated length is +larger than the header, the remaining bytes are payload. If the negotiated +length is 0 or unspecified, the packet length equals the header length (no +payload). + +The client fills the payload with zeros, random data, or a repeating +pattern, as configured. The server fills reply payloads according to its +configuration and the negotiated server fill parameter. + +### 8.2 Open Request Packet + +**Direction:** Client → Server +**Flags:** Open (0x01) set. Reply (0x02) clear. HMAC (0x08) set if HMAC key +configured. + +**Header fields present:** +- Magic (3 bytes) +- Flags (1 byte) +- HMAC (16 bytes) — if HMAC configured + +The connection token field is NOT present in the open request (the token has +not yet been assigned). [**Verified 2026-04-28** by packet capture — see +BLACKBOX_VERIFICATION_REPORT.md Section 19.1.] + +**Payload:** Serialized parameters (see Section 8.6). + +If the client wants to open and immediately close (no-test mode), both the +Open (0x01) and Close (0x04) flags are set. + +### 8.3 Open Reply Packet + +**Direction:** Server → Client +**Flags:** Open (0x01) set. Reply (0x02) set. Close (0x04) set if the server +is rejecting the connection. HMAC (0x08) set if applicable. + +**Header fields present:** +- Magic (3 bytes) +- Flags (1 byte) +- HMAC (16 bytes) — if applicable +- Connection Token (8 bytes) + +**Payload:** Serialized parameters (see Section 8.6), reflecting any server +restrictions. + +**Validation:** +- If the Close flag is NOT set, the connection token MUST be non-zero. A + zero token without the close flag is an error. +- If the Close flag IS set **and the client did NOT set Close in its open + request**, the server is rejecting the session (transition to Failed). +- If the Close flag IS set **and the client DID set Close in its open + request** (no-test mode), the server acknowledged the open+close. The + connection token is zero. Transition to Completed. + [**Verified 2026-04-28** — see BLACKBOX_VERIFICATION_REPORT.md Finding D.] + +### 8.4 Echo Request Packet (Test Packet) + +**Direction:** Client → Server +**Flags:** Reply (0x02) clear. HMAC (0x08) if applicable. Open (0x01) and +Close (0x04) clear. + +**Header fields present:** +- Magic (3 bytes) +- Flags (1 byte) +- HMAC (16 bytes) — if applicable +- Connection Token (8 bytes) +- Sequence Number (4 bytes) +- Received Count (4 bytes, zeroed) — if "count" stats negotiated +- Received Window (8 bytes, zeroed) — if "window" stats negotiated +- Timestamp fields (zeroed) — according to negotiated stamp_at and clock + +The echo request includes zeroed placeholder fields for received stats and +timestamps so that the total packet reaches the negotiated length. + +**Payload:** Filled according to client configuration (zeros, random, or +pattern). + +**Total packet length:** The negotiated `length` parameter. If the header +(including zeroed optional fields) is larger than the negotiated length, the +actual packet length is the header length. The final negotiated length +reflects this minimum. + +### 8.5 Echo Reply Packet + +**Direction:** Server → Client +**Flags:** Reply (0x02) set. HMAC (0x08) if applicable. Close (0x04) MAY be +set (see Section 19.3 — not yet verified by black-box testing). + +**Header fields present:** +- Magic (3 bytes) +- Flags (1 byte) +- HMAC (16 bytes) — if applicable +- Connection Token (8 bytes) +- Sequence Number (4 bytes) +- Received Count (4 bytes) — if negotiated, contains server's running count +- Received Window (8 bytes) — if negotiated, contains 64-bit received bitmap +- Timestamp fields — according to negotiated stamp_at and clock settings + +**Payload:** Server-filled according to negotiation. + +### 8.5.1 Close Request Packet + +**Direction:** Client → Server +**Flags:** Close (0x04) set. Reply (0x02) clear. HMAC (0x08) if applicable. + +**Header fields present:** +- Magic (3 bytes) +- Flags (1 byte) +- HMAC (16 bytes) — if applicable +- Connection Token (8 bytes) + +No sequence number, no payload. + +### 8.6 Parameter Serialization Format + +Parameters are serialized as a sequence of tag-value pairs in the packet +payload during the open request and open reply. + +Each parameter is encoded as: +1. **Tag:** Unsigned varint (LEB128 encoding). +2. **Value:** Signed varint (zigzag + LEB128 encoding), EXCEPT for + string-typed values. + +For string-typed values, the value is: +1. Length as unsigned varint. +2. Raw UTF-8 bytes. + +**Varint encoding:** This uses the same variable-length integer encoding as +Google Protocol Buffers: +- **Unsigned varint (uvarint):** Standard unsigned LEB128. Each byte uses + the high bit as a continuation flag. The remaining 7 bits contribute to + the value, least significant group first. +- **Signed varint (varint):** Uses zigzag encoding: `(v << 1) ^ (v >> 63)` + for encoding, decoded as `(uv >> 1) ^ -(uv & 1)`. This maps signed + integers to unsigned integers so that small-magnitude values (positive and + negative) have compact encodings. + +See Section 19.4 for cross-language verification notes. + +**Parameter tags:** + +| Tag | Name | Value Type | Unit / Encoding | Description | +|-----|------|-----------|-----------------|-------------| +| 1 | ProtocolVersion | Signed varint | Integer | Protocol version (currently 1) | +| 2 | Duration | Signed varint | Nanoseconds | Test duration | +| 3 | Interval | Signed varint | Nanoseconds | Send interval | +| 4 | Length | Signed varint | Bytes | Packet length | +| 5 | ReceivedStats | Signed varint | Enum (see below) | Server received packet statistics mode | +| 6 | StampAt | Signed varint | Enum (see below) | Server timestamp mode | +| 7 | Clock | Signed varint | Enum (see below) | Clock selection for timestamps | +| 8 | DSCP | Signed varint | Integer (TOS byte value) | IP TOS/Traffic Class byte value (see Section 10.8) | +| 9 | ServerFill | String (length-prefixed) | UTF-8, max 32 bytes | Server payload fill mode | + +Parameters with a zero value MAY be omitted from serialization (they will +default to zero on the receiving end). + +Unknown parameter tags MUST be silently ignored by the receiver. + +The maximum serialized parameter buffer size is believed to be 128 bytes +(see Section 19.10 for verification). + +**ReceivedStats enum values:** + +| Value | Name | Meaning | +|-------|------|---------| +| 0 | None | No received packet statistics | +| 1 | Count | Include received packet count | +| 2 | Window | Include 64-bit received window | +| 3 | Both | Include both count and window | + +**StampAt enum values:** + +| Value | Name | Meaning | +|-------|------|---------| +| 0 | None | No server timestamps | +| 1 | Send | Timestamp at server send | +| 2 | Receive | Timestamp at server receive | +| 3 | Both | Timestamps at both server receive and send | +| 4 | Midpoint | Midpoint timestamp (average of receive and send) | + +**Clock enum values:** + +| Value | Name | Meaning | +|-------|------|---------| +| 1 | Wall | Wall clock only | +| 2 | Monotonic | Monotonic clock only | +| 3 | Both | Both wall and monotonic clocks | + +--- + +## 9. Versioning and Compatibility + +### 9.1 Protocol Version + +The protocol version is negotiated during the open phase. The current (and +only) protocol version is **1**. + +A conforming client MUST set `ProtocolVersion` to 1 in its open request. + +A conforming client MUST verify that the server's returned protocol version +matches exactly. If there is a mismatch, the client MUST abort the session. +The server will also set the Close flag if there is a version mismatch. + +### 9.2 Unknown Fields and Extensions + +Unknown parameter tags in the negotiation payload MUST be silently ignored. + +Reserved flag bits (4-7) MUST NOT be set. Packets with unknown flag bits set +MUST be rejected. + +### 9.3 Server Versions + +This specification targets interoperability with servers implementing +protocol version 1, as found in IRTT versions 0.9.0 and later. Earlier +development versions (0.1.x) used an incompatible protocol and are not +supported. + +--- + +## 10. Client Configuration Parameters + +The following configuration parameters affect protocol behavior: + +### 10.1 Server Address + +- **Effect:** Determines the destination for all packets. +- **Default port:** 2112. +- **Valid values:** Hostname, IPv4 address, IPv6 address (in brackets), + optionally with port. + +### 10.2 Duration + +- **Effect:** How long the client sends test packets. +- **Default:** 60 seconds (1 minute). +- **Valid values:** Positive duration. +- **Protocol:** Sent as nanoseconds in the Duration parameter. The server + MAY reduce this value. + +### 10.3 Interval + +- **Effect:** Time between successive test packets. +- **Default:** 1 second. +- **Valid values:** Positive duration. +- **Protocol:** Sent as nanoseconds in the Interval parameter. The server + MAY increase this value (enforce minimum interval). The server MAY also + decrease it to stay within its timeout constraints. + +### 10.4 Length + +- **Effect:** Total UDP payload length of each test packet. +- **Default:** 0, which means the packet is the minimum size needed for the + header fields. +- **Valid values:** 0 or positive integer. The actual minimum is determined + by the header fields present. +- **Protocol:** Sent in the Length parameter. The server MAY reduce this. + +### 10.5 Received Stats + +- **Effect:** What statistics the server includes about received packets. +- **Default:** Both (count + window). +- **Valid values:** None (0), Count (1), Window (2), Both (3). + +### 10.6 Timestamp Mode (StampAt) + +- **Effect:** When the server records timestamps. +- **Default:** Both (send + receive). +- **Valid values:** None (0), Send (1), Receive (2), Both (3), Midpoint (4). +- **Note:** The server MAY restrict timestamps (e.g., allow only single or + no timestamps). + +### 10.7 Clock Selection + +- **Effect:** Which clock(s) are used for server timestamps. +- **Default:** Both (wall + monotonic). +- **Valid values:** Wall (1), Monotonic (2), Both (3). + +### 10.8 DSCP + +- **Effect:** Sets the TOS/Traffic Class byte in the IP header for both + client and server packets during the active test phase. +- **Default:** 0 (best effort). +- **Valid values:** 0-255 (full TOS byte value). The DSCP field occupies + the upper 6 bits; the lower 2 bits (ECN) are typically zero. + Common values: 0 (best effort), 0xb8 (EF), 0xa0 (CS5), 0x20 (CS1). + [**Verified 2026-04-28** — see BLACKBOX_VERIFICATION_REPORT.md Finding B.] +- **Protocol:** The server echoes packets with the negotiated DSCP value. + The server MAY disallow DSCP and reset it to 0. +- **Observed behavior:** DSCP/TOS is applied only to echo request and echo + reply packets. Open and close packets use TOS=0 regardless of the + negotiated DSCP value. [**Verified 2026-04-28** — Finding C.] + +### 10.9 Open Timeouts + +- **Effect:** Controls retransmission during the open phase. +- **Default:** 1s, 2s, 4s, 8s. +- **Valid values:** List of positive durations, each >= 200ms. + +### 10.10 Wait (Drain Period) + +- **Effect:** How long to wait after the last send for outstanding replies. +- **Default:** 3 times the maximum observed RTT, falling back to 4 seconds + if no RTT has been measured. +- **Variants:** + - A factor of maximum observed RTT, with a fallback duration if no RTT + is available. + - A factor of mean observed RTT, with a fallback duration if no RTT is + available. + - A fixed duration. + +### 10.11 HMAC Key + +- **Effect:** When set, an MD5-based HMAC is computed over each packet and + included in the HMAC field. The server MUST also have the same key. +- **Default:** None (no HMAC). +- **Protocol:** When an HMAC key is configured, the HMAC flag (0x08) is set + in every packet. + +### 10.12 Server Fill + +- **Effect:** Requests the server to fill reply payloads with specific data. +- **Default:** Not specified (server uses its default fill). +- **Valid values:** String, max 32 characters. Common values: "none", "rand", + "pattern:XXXX" (hex pattern). +- **Protocol:** The server MAY restrict this to allowed fill types. + +### 10.13 No-Test Mode + +- **Effect:** Opens a connection and immediately closes it without running a + test. Useful for parameter validation. +- **Protocol:** Both the Open and Close flags are set in the open request. + +### 10.14 Loose Mode + +- **Effect:** When enabled, the client accepts server-restricted parameters + with a warning instead of aborting. +- **Default:** Disabled (strict mode). + +--- + +## 11. Timing Semantics + +### 11.1 Timestamp Capture Points + +- **Client send timestamp:** SHOULD be captured as close as possible to the + send system call. For best RTT accuracy, the timestamp should be taken + immediately after the send call returns. +- **Client receive timestamp:** SHOULD be captured as close as possible to + the receive system call return, before any packet processing. +- **Server timestamps:** Determined by the negotiated StampAt parameter: + - Receive: captured when the server receives the packet. + - Send: captured just before the server sends the reply. + - Midpoint: the average of receive and send times. + +### 11.2 Clock Types + +A conforming client MUST track both wall clock and monotonic clock times +for client-side timestamps. The monotonic clock SHOULD be used for all +duration calculations (RTT, IPDV) because it is immune to system time +adjustments. The wall clock is needed for one-way delay calculations. + +### 11.3 Send Scheduling + +The client sends packets on a **fixed isochronous schedule**: + +1. Record the start time. +2. The ideal send time for packet N is: `start + N * interval`. +3. The client SHOULD send each packet as close to its ideal send time as + the system timer allows. +4. The client SHOULD compensate for cumulative timer drift so that packets + remain aligned to integer multiples of the interval from the start time. + The specific drift-compensation strategy is implementation-defined. +5. Before each send, the client checks whether the next scheduled time + would be at or past `start + duration`. If so, sending stops. + +The observable effect is that packets are sent at approximately +`start + 0`, `start + interval`, `start + 2*interval`, etc., with the +last packet sent strictly before `start + duration`. + +### 11.4 Duration and Count Interaction + +- The expected packet count is `ceil(duration / interval)`. + [**Verified 2026-04-28** — see BLACKBOX_VERIFICATION_REPORT.md Finding A.] +- The duration is **exclusive**: the last packet is sent at approximately + `start + (ceil(duration / interval) - 1) * interval`, which is strictly + before `start + duration`. +- Example: duration=1s, interval=200ms → packets at 0ms, 200ms, 400ms, + 600ms, 800ms = 5 packets = ceil(1000/200). +- Example: duration=1s, interval=1s → packet at 0ms only = 1 packet = + ceil(1000/1000). The packet at 1000ms is NOT sent (exclusive end). + +### 11.5 Timer Compensation + +The client SHOULD use timer error compensation to improve send accuracy. +The specific compensation algorithm is implementation-defined. Any approach +that produces accurate isochronous sending is acceptable. + +### 11.6 Drain Period + +After the last packet is sent, if not all replies have been received, the +client MUST wait for outstanding replies for the configured wait duration +before closing the session. + +--- + +## 12. Measurement Semantics + +### 12.1 Round-Trip Time (RTT) + +RTT MUST be calculated using client-side monotonic clock values: + +``` +RTT = client_receive_mono - client_send_mono +``` + +If both server send and receive timestamps are available (monotonic +preferred, wall clock as fallback), the server processing time SHOULD be +subtracted: + +``` +server_processing_time = server_send - server_receive +RTT = (client_receive_mono - client_send_mono) - server_processing_time +``` + +This subtraction SHOULD only be applied when both server timestamps are +available. See Section 19.12 for edge-case behavior when server processing +time appears to exceed the raw round-trip time. + +### 12.2 One-Way Delay + +One-way delay requires wall clock timestamps and externally synchronized +clocks (e.g., NTP, PTP). Values are only meaningful when clocks are +synchronized. + +``` +send_delay = server_best_receive_wall - client_send_wall +receive_delay = client_receive_wall - server_best_send_wall +``` + +Where "best receive" prefers the actual receive timestamp, falling back to +the send timestamp if receive is not available (and vice versa for "best +send"). For midpoint timestamps, both best_receive and best_send are the +midpoint value. + +### 12.3 Packet Loss + +``` +packet_loss_percent = 100 * (packets_sent - unique_replies_received) / packets_sent +``` + +If the received count is available from the server: + +``` +upstream_loss_percent = 100 * (packets_sent - server_packets_received) / packets_sent +downstream_loss_percent = 100 * (server_packets_received - unique_replies_received) / server_packets_received +``` + +### 12.4 Duplicate Packets + +A duplicate is a reply with a sequence number for which a reply has already +been recorded. Duplicates MUST be counted but MUST NOT update the original +round-trip data for that sequence number. + +``` +duplicate_percent = 100 * duplicates / total_packets_received +``` + +### 12.5 Late Packets (Out-of-Order) + +A late packet is one whose sequence number is lower than the most recently +received sequence number. + +``` +late_packets_percent = 100 * late_packets / total_packets_received +``` + +### 12.6 IPDV (Jitter) + +IPDV is the difference in delay between two successive successfully received +replies. It is only calculable for sequence number N when both packet N and +packet N-1 have received replies. + +``` +round_trip_ipdv[N] = RTT[N] - RTT[N-1] +send_ipdv[N] = send_delay[N] - send_delay[N-1] (if timestamps available) +receive_ipdv[N] = receive_delay[N] - receive_delay[N-1] (if timestamps available) +``` + +For send and receive IPDV, monotonic clock differences are RECOMMENDED +over wall clock differences when both are available, as the monotonic clock +is not subject to NTP adjustments. Note that IPDV does NOT require +synchronized clocks — only consistent clocks within each endpoint. + +### 12.7 Server Processing Time + +When both server send and receive timestamps are available: + +``` +server_processing_time = server_send - server_receive +``` + +Prefer monotonic clock values if available; fall back to wall clock. + +### 12.8 Per-Packet Upstream/Downstream Loss (Received Window) + +When the 64-bit received window is present in a reply: +- Bit 0 (LSB) represents the current packet (the one being replied to). +- Bit 1 represents the packet with sequence number one less than current, etc. +- A set bit means the server received that packet; a clear bit means it + did not. + +The window may not be valid for all replies. Bit 0 (LSB) is always set +for valid windows, since it represents the current packet which was +received. [**Verified 2026-04-28** — see Section 19.7.] Implementations +SHOULD treat a window value of 0 as potentially invalid and avoid using +it for loss classification. + +The client processes the received window to classify previous packets: +- If a packet was not received by the client but the window says the server + received it → lost downstream. +- If a packet was not received by the client and the window says the server + did not receive it → lost upstream. +- If the window cannot confirm (e.g., more than 64 packets ago, or window + was invalid) → generic loss. + +The window has a maximum lookback of 63 packets (bits 1-63). + +### 12.9 Statistics + +For RTT, send delay, receive delay, and IPDV, a conforming client SHOULD +calculate: +- Minimum, Maximum, Mean, Median, Standard Deviation, Variance. +- Total count (N). + +Median requires storing all values. The method of computing running +statistics is implementation-defined. + +--- + +## 13. Validation Rules + +### 13.1 Magic Bytes + +Every received packet MUST begin with the magic bytes `0x14 0xA7 0x5B`. +Packets with incorrect magic MUST be discarded. + +### 13.2 Flag Validation + +Reserved flag bits (4-7) MUST be zero. Packets with invalid flag bits MUST +be discarded. + +### 13.3 Reply Flag + +All packets received by the client MUST have the Reply flag (0x02) set. +Packets without the Reply flag MUST be discarded. + +### 13.4 HMAC Validation + +If an HMAC key is configured: +- Received packets MUST have the HMAC flag set. +- The HMAC MUST be validated: + 1. Extract the HMAC value from the packet. + 2. Zero the HMAC field in the packet buffer. + 3. Compute HMAC-MD5 over the entire packet buffer. + 4. Compare the computed HMAC with the extracted value using a + constant-time comparison. +- Packets with missing or invalid HMAC MUST be discarded. + +If no HMAC key is configured: +- Received packets MUST NOT have the HMAC flag set. Packets with an + unexpected HMAC MUST be discarded. + +### 13.5 Sequence Number Validation + +Echo reply sequence numbers MUST correspond to a previously sent packet. +Replies with sequence numbers outside the range [0, packets_sent) MUST be +rejected. + +### 13.6 Timestamp Consistency + +If both receive and send timestamps are present in a reply, they MUST use +the same clock mode (both wall, both monotonic, or both sets). Inconsistent +clock modes MUST cause the packet to be rejected. + +Midpoint timestamps MUST be exclusive — if a midpoint timestamp is present, +no receive or send timestamp fields may also be present. + +### 13.7 Close Flag in Echo Reply + +If the server sets the Close flag in an echo reply, the client SHOULD +process the reply normally (record measurements) and then close the +connection. This is defensive handling for the possibility that the +server can forcibly end a session mid-test. + +**Note:** This behavior has not been directly verified by black-box +testing (see Section 19.3). Implementations SHOULD handle it defensively. + +--- + +## 14. Error Handling + +| Error Category | Client Behavior | +|---------------|----------------| +| Name resolution failure | MUST abort. | +| Socket creation / bind failure | MUST abort. | +| Server unreachable (all open timeouts exhausted) | MUST abort. No partial results. | +| Protocol version mismatch | MUST abort. | +| Server restriction rejected (strict mode) | MUST abort. | +| Server restriction accepted (loose mode) | SHOULD warn, continue. | +| DSCP not supported by OS | MUST abort if DSCP was requested non-zero. | +| DF bit not supported by OS | MUST abort if DF was requested non-default. | +| Open reply with Close flag (not no-test) | MUST abort (server rejected session). | +| Zero connection token (without close flag) | MUST abort. | +| Send error during test | SHOULD abort, report partial results. | +| Receive error during test | SHOULD abort, report partial results. | +| Individual malformed packet during test | SHOULD abort. | +| Short reply (shorter than negotiated length) | SHOULD abort. | +| Stamp-at mismatch in reply | SHOULD abort. | +| Clock mismatch in reply | SHOULD abort. | +| Unexpected sequence number | SHOULD abort. | +| Context cancellation (user interrupt) | SHOULD stop gracefully, close connection, report partial results. | +| Insufficient resources for test parameters | MUST abort. | + +--- + +## 15. Result Model + +After a test, the client implementation SHOULD expose the following +information: + +### 15.1 Per-Packet Data (Round Trips) + +For each sent packet (indexed by sequence number): +- Sequence number. +- Lost status: not lost, lost (generic), lost upstream, lost downstream. +- Client send timestamp (wall + monotonic). +- Client receive timestamp (wall + monotonic) — empty if lost. +- Server receive timestamp (wall and/or monotonic) — as negotiated. +- Server send timestamp (wall and/or monotonic) — as negotiated. +- Computed RTT. +- Computed send delay. +- Computed receive delay. +- Computed IPDV (round-trip, send, receive) relative to previous packet. + +### 15.2 Aggregate Statistics + +- Test start time (wall + monotonic). +- Actual test duration. +- Packets sent. +- Packets received (unique). +- Packet loss percentage. +- Upstream loss percentage (if server count available). +- Downstream loss percentage (if server count available). +- Duplicate count and percentage. +- Late packet count and percentage. +- RTT statistics: min, max, mean, median, stddev, variance. +- Send delay statistics (if timestamps available). +- Receive delay statistics (if timestamps available). +- IPDV statistics (round-trip, send, receive). +- Server processing time statistics. +- Bytes sent and received. +- Send and receive bitrate. +- Timer error statistics. +- Timer miss count and percentage. +- Wait duration actually used. + +### 15.3 Configuration + +- Local and remote addresses (resolved). +- Negotiated parameters. +- Originally supplied parameters. +- IP version used. + +### 15.4 Errors + +- Send error (if the send operation failed). +- Receive error (if the receive operation failed). + +--- + +## 16. Interoperability Requirements + +1. A conforming client MUST be able to open a session with a compatible IRTT + server running protocol version 1. + +2. A conforming client MUST send open requests with correctly serialized + parameters using the varint-encoded tag-value format. + +3. A conforming client MUST correctly parse server open replies, including + restricted parameters. + +4. A conforming client MUST include the connection token in all post-open + packets. + +5. A conforming client MUST send test packets in the negotiated format, with + the correct field layout and packet length. + +6. A conforming client MUST correctly match replies to sent packets using + sequence numbers. + +7. A conforming client MUST tolerate UDP reordering, duplication, and loss. + +8. A conforming client MUST validate magic bytes, flags, and HMAC on all + received packets. + +9. A conforming client SHOULD produce RTT measurements consistent with + upstream implementations within the precision of the system clocks. + +10. A conforming client MUST send a close packet at the end of the session. + +11. A conforming client MUST correctly handle the HMAC calculation: + compute HMAC-MD5 over the full packet with the HMAC field zeroed. + +12. When computing an outgoing HMAC, the client MUST: + a. Set the HMAC flag bit. + b. Zero the HMAC field. + c. Compute HMAC-MD5 over the entire packet buffer. + d. Write the resulting MAC into the HMAC field. + +--- + +## 17. Black-Box Test Plan + +### Test 1: Basic Connectivity + +- **Setup:** Start an upstream IRTT server with default settings. +- **Client config:** Default parameters (1s interval, 1m duration), reduce + to 5s duration for testing. Server address: localhost. +- **Expected:** Session opens, packets are exchanged, results include RTT + measurements, session closes cleanly. +- **Pass criteria:** Non-zero packets_sent and packets_received. RTT values + present. No errors. + +### Test 2: Parameter Negotiation + +- **Setup:** Start a server with `--max-length=100`. +- **Client config:** Request length 200. +- **Expected (strict mode):** Client aborts with server restriction error. +- **Expected (loose mode):** Client continues with length 100. +- **Pass criteria (strict):** Client exits with error indicating server + reduced length. +- **Pass criteria (loose):** Successful test with negotiated length 100. + +### Test 3: HMAC Authentication + +- **Setup:** Start server with `--hmac=testkey`. +- **Client config:** `--hmac=testkey`. +- **Expected:** Session succeeds. +- **Pass criteria:** Successful test with no errors. +- **Variant:** Client uses wrong key. Expected: open timeout (server drops + packets with bad HMAC silently). + +### Test 4: Packet Capture Verification + +- **Setup:** Start server on localhost. Capture with tcpdump on the loopback + interface, port 2112. +- **Client config:** Short test (5s, 1s interval). +- **Expected:** Capture shows: + - Open request: magic bytes 14 a7 5b, flags=0x01, followed by varint + params in payload. + - Open reply: magic bytes, flags=0x03 (open+reply), 8-byte token, params. + - Echo requests: magic bytes, flags=0x00, 8-byte token, 4-byte sequence number. + - Echo replies: magic bytes, flags=0x02 (reply), 8-byte token, 4-byte + sequence number, optional stats and timestamps. + - Close request: magic bytes, flags=0x04. +- **Pass criteria:** All packets have correct magic, flags, and field layout. + +### Test 5: Sequence Number Handling + +- **Setup:** Server with default settings. +- **Client config:** 10s duration, 200ms interval (expect ~51 packets). +- **Expected:** Reply sequence numbers are in range [0, packets_sent). + Results show correct per-packet data. +- **Pass criteria:** All received replies have valid sequence numbers. No + unexpected sequence number errors. + +### Test 6: Timestamp Modes + +- **Setup:** Server with default settings. +- **Variants:** + - `--tstamp=none --clock=wall` + - `--tstamp=send --clock=wall` + - `--tstamp=receive --clock=monotonic` + - `--tstamp=both --clock=both` + - `--tstamp=midpoint --clock=both` +- **Expected:** Each variant produces the correct timestamp fields in replies. + RTT is always available. One-way delay is only available with wall clock + timestamps. +- **Pass criteria:** Results match expected availability of delay measurements. + +### Test 7: Loss Detection + +- **Setup:** Use a network impairment tool (e.g., `tc netem loss 10%`) or a + server on a lossy link. +- **Client config:** 30s duration, 100ms interval. +- **Expected:** packet_loss_percent is non-zero. With `--stats=both`, + upstream and downstream loss are differentiated. +- **Pass criteria:** Loss percentages are reported. With received window, + per-packet loss direction is classified. + +### Test 8: IPv6 + +- **Setup:** Server listening on IPv6 (::1). +- **Client config:** `-6 ::1`. +- **Expected:** Successful test over IPv6. +- **Pass criteria:** IP version in results is IPv6. + +### Test 9: No-Test Mode + +- **Setup:** Server with default settings. +- **Client config:** `-n localhost`. +- **Expected:** Connection opens, parameters are negotiated, connection + closes immediately. No test packets sent. +- **Pass criteria:** No round-trip data. No errors. + +### Test 10: Large Packet Size + +- **Setup:** Server with default settings. +- **Client config:** `-l 1472 -d 5s localhost`. +- **Expected:** Successful test with 1472-byte packets. +- **Pass criteria:** Non-zero results. + +### Test 11: Server Fill + +- **Setup:** Server with `--allow-fills=rand`. +- **Client config:** `--sfill=rand -l 172 -d 5s localhost`. +- **Expected:** Successful test. Server fills reply payloads with random data. +- **Pass criteria:** Test completes without error. + +### Test 12: DSCP + +- **Setup:** Server with default settings (DSCP allowed). +- **Client config:** `--dscp=46 -d 5s localhost`. +- **Expected:** Packets are sent with EF DSCP marking. Capture verification + shows TOS/TC field set. +- **Pass criteria:** Test completes. Packet capture confirms DSCP value. + +--- + +## 18. Packet Test Vectors + +### 18.1 Echo Request with HMAC + +**Scenario:** Echo request with HMAC, received stats (both), timestamps +(both, both clocks), pattern fill `0xFF 0xFE 0xFD 0xFC`, 92-byte packet +(76-byte header + 16-byte payload). +[**Size corrected 2026-04-28** — originally described as 256 bytes, but +HMAC verification confirms the test vector is 92 bytes. See +BLACKBOX_VERIFICATION_REPORT.md Finding E.] + +**Connection token:** `0x886bc9a722b33eea` (little-endian) +**Sequence number:** `0x6fe2a1bb` (little-endian) +**HMAC key:** `0x3c 0x68 0x1d 0x39 0x41 0x1d 0x72 0x43` + +**Fields present (in order):** +1. Magic: 3 bytes +2. Flags: 1 byte (0x08 = HMAC) +3. HMAC: 16 bytes +4. Connection Token: 8 bytes +5. Sequence Number: 4 bytes +6. Received Count: 4 bytes (zeroed) +7. Received Window: 8 bytes (zeroed) +8. Receive Wall: 8 bytes (zeroed) +9. Receive Mono: 8 bytes (zeroed) +10. Send Wall: 8 bytes (zeroed) +11. Send Mono: 8 bytes (zeroed) + +**Header total:** 3+1+16+8+4+4+8+8+8+8+8 = 76 bytes +**Payload:** 92 - 76 = 16 bytes of repeating pattern + +**Hex (92 bytes):** +``` +14 a7 5b 08 e7 03 41 e7 d4 08 cf 69 41 f3 f4 78 +5a 56 0c 4c ea 3e b3 22 a7 c9 6b 88 bb a1 e2 6f +00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 +00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 +00 00 00 00 00 00 00 00 00 00 00 00 ff fe fd fc +ff fe fd fc ff fe fd fc ff fe fd fc +``` + +**Field breakdown:** +- Bytes 0-2: Magic `14 a7 5b` +- Byte 3: Flags `08` (HMAC set) +- Bytes 4-19: HMAC `e7 03 41 e7 d4 08 cf 69 41 f3 f4 78 5a 56 0c 4c` +- Bytes 20-27: Connection Token `ea 3e b3 22 a7 c9 6b 88` (LE: 0x886bc9a722b33eea) +- Bytes 28-31: Sequence Number `bb a1 e2 6f` (LE: 0x6fe2a1bb) +- Bytes 32-35: Received Count (zeroed) +- Bytes 36-43: Received Window (zeroed) +- Bytes 44-51: Receive Wall Timestamp (zeroed) +- Bytes 52-59: Receive Mono Timestamp (zeroed) +- Bytes 60-67: Send Wall Timestamp (zeroed) +- Bytes 68-75: Send Mono Timestamp (zeroed) +- Bytes 76+: Payload (pattern fill) + +**Client behavior:** This packet is sent to the server during the active test phase. + +### 18.2 Echo Reply with HMAC and Full Timestamps + +**Scenario:** Echo reply with HMAC, received stats (both), timestamps (both, +both clocks), pattern fill, 92-byte packet (76-byte header + 16-byte payload). +[**Size corrected 2026-04-28** — see Finding E.] + +**Connection token:** `0xe666ceb6766fcbc3` (little-endian) +**Sequence number:** `0x1d3b0706` (little-endian) +**HMAC key:** `0xda 0xb3 0xe9 0x04 0xa6 0x87 0x92 0x49` +**Received count:** `0xa3bc9f19` (little-endian) +**Received window:** `0xd7e939e586f83b9b` (little-endian) + +**Timestamps (little-endian int64):** +- Receive wall: `0x525af13dee2a75a1` +- Receive mono: `0x2d5562223e4ac69a` +- Send wall: `0x589705d446293f69` +- Send mono: `0x461df12fdd2c5066` + +**Fields present (in order):** +1. Magic: 3 bytes +2. Flags: 1 byte (0x08 = HMAC, plus 0x02 = Reply → but let's verify from the test vector) +3. HMAC: 16 bytes +4. Connection Token: 8 bytes +5. Sequence Number: 4 bytes +6. Received Count: 4 bytes +7. Received Window: 8 bytes +8. Receive Wall: 8 bytes +9. Receive Mono: 8 bytes +10. Send Wall: 8 bytes +11. Send Mono: 8 bytes + +**Note on flags:** The flags byte in this test vector is `0x08` (only HMAC +set). This vector is provided for verifying field layout and HMAC +computation only. **In actual protocol operation, the Reply flag (0x02) +MUST also be set on all server-to-client packets**, making the expected +flags byte `0x0A` for a normal HMAC echo reply. Implementers MUST +set the Reply flag in actual protocol exchanges; the HMAC computation +itself does not depend on the Reply flag's value. + +**Hex (first 92 bytes):** +``` +14 a7 5b 08 d2 98 a3 4a 6a 13 41 02 68 b2 67 a8 +d6 7e 28 25 c3 cb 6f 76 b6 ce 66 e6 06 07 3b 1d +19 9f bc a3 9b 3b f8 86 e5 39 e9 d7 a1 75 2a ee +3d f1 5a 52 9a c6 4a 3e 22 62 55 2d 69 3f 29 46 +d4 05 97 58 66 50 2c dd 2f f1 1d 46 ff fe fd fc +ff fe fd fc ff fe fd fc ff fe fd fc +``` + +**Field breakdown:** +- Bytes 0-2: Magic `14 a7 5b` +- Byte 3: Flags `08` (HMAC; see note above about Reply flag) +- Bytes 4-19: HMAC `d2 98 a3 4a 6a 13 41 02 68 b2 67 a8 d6 7e 28 25` +- Bytes 20-27: Connection Token `c3 cb 6f 76 b6 ce 66 e6` (LE: 0xe666ceb6766fcbc3) +- Bytes 28-31: Sequence Number `06 07 3b 1d` (LE: 0x1d3b0706) +- Bytes 32-35: Received Count `19 9f bc a3` (LE: 0xa3bc9f19) +- Bytes 36-43: Received Window `9b 3b f8 86 e5 39 e9 d7` (LE: 0xd7e939e586f83b9b) +- Bytes 44-51: Receive Wall `a1 75 2a ee 3d f1 5a 52` (LE: 0x525af13dee2a75a1) +- Bytes 52-59: Receive Mono `9a c6 4a 3e 22 62 55 2d` (LE: 0x2d5562223e4ac69a) +- Bytes 60-67: Send Wall `69 3f 29 46 d4 05 97 58` (LE: 0x589705d446293f69) +- Bytes 68-75: Send Mono `66 50 2c dd 2f f1 1d 46` (LE: 0x461df12fdd2c5066) +- Bytes 76+: Payload (pattern fill) + +**Expected client behavior:** Validate magic, validate HMAC, extract +connection token, extract sequence number, extract received stats, extract timestamps, +record round-trip data. + +--- + +## 19. Open Questions / Verification Required + +### 19.1 Open Request Field Layout — ✅ RESOLVED + +**Status:** Verified 2026-04-28, irtt 0.9.1, macOS arm64. + +**Result:** The open request does NOT include a connection token field. The +layout is: magic (3) + flags (1) + [HMAC (16) if applicable] + params. +Verified by packet capture (`captures/full-session.pcapng`, +`captures/hmac-session.pcapng`). + +### 19.2 Minimum Packet Length Calculation — ✅ RESOLVED + +**Status:** Verified 2026-04-28, irtt 0.9.1, macOS arm64. + +**Result:** Minimum echo packet lengths verified for all stats/tstamp/clock +combinations. See BLACKBOX_VERIFICATION_REPORT.md Section 19.2 for the +complete table. The minimum is 16 bytes (no stats, no timestamps) and the +maximum default is 76 bytes (both+both+both+HMAC). + +### 19.3 Server Close During Test — OPEN / Verification Required + +**Status:** Not tested. Hard to trigger reliably with black-box testing. + +**Question:** Can the server set the Close flag on an echo reply mid-test +(e.g., due to exceeding a duration limit)? If so, how should the client +respond? + +**Compatibility recommendation:** The client SHOULD check for the Close +flag in echo replies and, if set, process the reply normally (record +measurements) and then terminate the session. This is defensive handling +for a scenario that has not been directly observed in black-box testing. + +### 19.4 Varint Encoding Compatibility — ✅ RESOLVED + +**Status:** Verified 2026-04-28, irtt 0.9.1, macOS arm64. + +**Result:** Parameters decoded from captured packets match expected values +using standard protobuf-style zigzag + LEB128. All parameter tags and values +verified. See BLACKBOX_VERIFICATION_REPORT.md Section 19.4. + +### 19.5 HMAC Computation Scope — ✅ RESOLVED + +**Status:** Verified 2026-04-28, irtt 0.9.1, macOS arm64. + +**Result:** HMAC-MD5 is computed over the entire packet buffer (including +payload) with the 16-byte HMAC field zeroed. Independently verified with +Python for all four packet types (open request, echo request, echo reply, +close request). The spec's test vectors (18.1 and 18.2) also verify +correctly at 92-byte packet size. + +### 19.6 Send Timestamp Capture Timing — OPEN (not externally testable) + +This is an implementation-internal detail that cannot be verified via +black-box testing. The client SHOULD capture timestamps as close to the +send/receive system calls as possible (see Section 11.1), but the exact +timing is an implementation choice. + +### 19.7 Server Received Window Validity — ✅ RESOLVED + +**Status:** Verified 2026-04-28, irtt 0.9.1, macOS arm64. + +**Result:** Bit 0 (LSB) of the received window is always set for valid +windows (it represents the current packet which was received). Observed +window values: seq 0 → 0x01, seq 1 → 0x03, seq 2 → 0x07. A window +value of 0 can be used as the invalid sentinel. See +BLACKBOX_VERIFICATION_REPORT.md Section 19.7. + +### 19.8 Open Reply Without HMAC When Client Expects HMAC — ✅ RESOLVED + +**Status:** Verified 2026-04-28, irtt 0.9.1, macOS arm64. + +**Result:** The server silently drops packets with missing or invalid HMAC. +The client times out with `[OpenTimeout] no reply from server`. Same +behavior for wrong key and no key. See BLACKBOX_VERIFICATION_REPORT.md +Section 19.8. + +### 19.9 Minimum Restricted Interval Safety Floor — PARTIALLY RESOLVED + +**Status:** Partially tested 2026-04-28. + +**Result:** In strict mode, the client rejects ANY server parameter change +with a non-zero exit code. In loose mode, changes are accepted with a +warning. The exact safety floor for server-decreased intervals could not +be tested without a cooperative malicious server. See +BLACKBOX_VERIFICATION_REPORT.md Section 19.9. + +### 19.10 Maximum Parameter Buffer Size — PARTIALLY RESOLVED + +**Status:** Partially tested 2026-04-28. + +**Result:** Largest observed parameter payload was 52 bytes (all params +including 24-character ServerFill). Upper bound estimate is ~82 bytes. The +128-byte limit appears safe. See BLACKBOX_VERIFICATION_REPORT.md +Section 19.10. + +### 19.11 Field Ordering Verification — ✅ RESOLVED + +**Status:** Verified 2026-04-28, irtt 0.9.1, macOS arm64. + +**Result:** The field ordering in Section 8.1.3 is correct. Verified with +six different field combinations including no optional fields, various +stats/tstamp combinations, midpoint timestamps, and HMAC. See +BLACKBOX_VERIFICATION_REPORT.md Section 19.11. + +### 19.12 RTT When Server Processing Time Exceeds Raw Round-Trip — OPEN / Verification Required + +**Status:** Not tested. Hard to trigger on localhost. + +**Question:** When server processing time (as reported by server +timestamps) exceeds the client-measured round-trip time — possible due to +clock granularity or drift — the subtraction would produce a negative RTT. +Should the client skip the subtraction, clamp to zero, or report the +negative value? + +This edge case requires server processing time to exceed the raw +client-measured RTT, which is extremely unlikely under normal conditions. + +--- + +## 20. Non-Goals + +This specification does **not** require: + +- Matching upstream source layout, file organization, or module structure. +- Matching upstream private names (functions, types, variables). +- Matching upstream internal algorithms where equivalent behavior can be + achieved independently. +- Matching exact CLI text output. +- Implementing the server. +- Implementing every optional upstream feature in the first milestone. +- Implementing timer compensation with any specific algorithm (only accurate + send timing is required). +- Implementing any specific concurrency model. +- Using any specific programming language features or libraries. +- Supporting IRTT protocol versions other than 1. +- Supporting the development-era (0.1.x) protocol. diff --git a/docs/RUST_DESIGN.md b/docs/RUST_DESIGN.md new file mode 100644 index 0000000..3288c7a --- /dev/null +++ b/docs/RUST_DESIGN.md @@ -0,0 +1,1452 @@ +# IRTT-RS Rust Design Draft + +## Status + +This document is a Rust-side architecture and implementation design brief for an independent IRTT-compatible client suite. + +It is intended to be consumed by a clean implementation-planning or implementation agent together with: + +- `IRTT_CLIENT_PROTOCOL_SPEC.md` +- `BLACKBOX_VERIFICATION_REPORT.md` +- clean packet test vectors +- clean-room process notes + +This document describes Rust architecture and product goals. It is not a protocol specification and does not copy or depend on upstream source structure. + +--- + +## Core Product Goal + +Build a layered, low-memory, event-producing IRTT-compatible Rust client suite. + +The project is not primarily an upstream CLI clone. The primary design goal is a reusable client engine suitable for: + +- `sqm-autorate-rust` +- `cake-autorate` integration through a CLI +- router-friendly continuous latency probing +- normal user-facing finite IRTT-style tests +- optional aggregate statistics and reports + +The central architectural rule is: + +```text +irtt-client emits probe events. +irtt-stats aggregates probe events. +irtt-cli formats probe events and/or statistics. +``` + +Aggregate statistics must not be welded into the client engine. + +--- + +## License + +Preferred project license: + +```text +Apache-2.0 +``` + +Optional alternative if broader Rust ecosystem convention is desired: + +```text +MIT OR Apache-2.0 +``` + +The implementation agent must not inspect upstream GPL source code, upstream tests, upstream comments, contaminated notes, or contaminated transcripts. + +--- + +## Workspace Layout + +Use a Cargo workspace with four crates: + +```text +irtt-rs/ + Cargo.toml + crates/ + irtt-proto/ + irtt-client/ + irtt-stats/ + irtt-cli/ + docs/ + IRTT_CLIENT_PROTOCOL_SPEC.md + BLACKBOX_VERIFICATION_REPORT.md + CLEANROOM.md + RUST_DESIGN.md + test-vectors/ +``` + +--- + +# Crate Responsibilities + +## `irtt-proto` + +Pure protocol crate. + +### Responsibilities + +`irtt-proto` owns only binary/protocol logic: + +- packet flags +- magic/version constants +- varint/zigzag parameter encoding and decoding +- parameter model +- packet layout calculation +- open request/reply encoding and decoding +- echo request/reply encoding and decoding +- close request encoding +- HMAC-MD5 calculation and verification +- clean packet vector tests + +### Must Not Contain + +`irtt-proto` must not contain: + +- sockets +- threads +- channels +- timers +- CLI code +- statistics aggregation +- session runtime control +- event fanout +- long-term sample retention +- OS-specific network metadata handling + +### Important Boundary + +ECN, DSCP, TOS, Traffic Class, and kernel receive timestamps are socket/IP metadata. They are not IRTT payload fields. `irtt-proto` must not model ECN as a protocol field. + +--- + +## `irtt-client` + +Client engine crate. + +### Responsibilities + +`irtt-client` owns: + +- connected UDP socket setup +- socket configuration through `socket2` +- session negotiation +- open/no-test/active/close lifecycle +- caller-paced manual client API +- optional managed threaded runner +- probe sending +- reply receiving +- packet validation +- timestamp capture +- RTT sample construction +- duplicate detection +- late detection +- loss timeout detection +- public event types +- optional public event fanout for managed mode + +### Must Not Contain + +`irtt-client` must not contain: + +- aggregate statistics +- rolling windows +- median/stddev/variance +- human final reports +- CLI formatting +- mandatory full sample history +- upstream-style result monoliths + +### Retention Boundary + +`irtt-client` must not retain samples for reporting or aggregation. + +It may retain bounded in-flight operational state required for: + +- matching replies to sent probes +- computing RTT for completed probes +- detecting duplicates +- detecting late replies +- detecting loss after timeout +- managing sequence/session state + +Historical retention, rolling windows, finite-run summaries, and aggregate statistics belong in `irtt-stats` or in the consuming application. + +--- + +## `irtt-stats` + +Optional statistics crate. + +### Responsibilities + +`irtt-stats` consumes `ClientEvent` values and provides: + +- cumulative statistics +- rolling time-window statistics +- rolling sample-count statistics +- finite-run summaries +- RTT summaries +- raw RTT summaries +- adjusted RTT summaries +- IPDV summaries +- server-processing summaries +- loss summaries +- duplicate/late summaries +- min/max/mean/median/stddev/variance where supported + +### Must Not Contain + +`irtt-stats` must not contain: + +- sockets +- packet encoding +- packet decoding +- session negotiation +- thread/runtime ownership +- CLI parsing + +It is a consumer of events, not a driver of the client. + +--- + +## `irtt-cli` + +User-facing executable crate. + +### Long-Term Role + +`irtt-cli` should become the full user-facing client over time. + +It should eventually support practical feature completeness for client usage: + +- finite tests +- continuous streaming +- human output +- machine output +- statistics +- HMAC +- DSCP / Traffic Class +- timestamp modes +- received stats modes +- server fill +- no-test/check mode +- optional JSON output + +### Stats Relationship + +`irtt-cli` should consume `irtt-stats` by default in normal builds. + +The stats dependency is feature-gated for binary size and resource usage, not because stats are second-class. + +Default/full CLI builds should include stats. + +Minimal/router builds may disable stats to produce a smaller streaming-only binary. + +```toml +[features] +default = ["stats"] +stats = ["dep:irtt-stats"] +json = ["dep:serde_json", "irtt-client/serde", "irtt-stats/serde"] +tracing = ["irtt-client/tracing"] +ancillary = ["irtt-client/ancillary"] +interop = [] +``` + +Minimal build example: + +```bash +cargo build -p irtt-cli --no-default-features --release +``` + +Normal build example: + +```bash +cargo build -p irtt-cli --release +``` + +### Layering Rule + +`irtt-cli` must compose `irtt-client` and `irtt-stats`. It must not reimplement protocol, session, or statistics logic internally. + +--- + +# Public Client Layers + +`irtt-client` should provide two public client layers: + +1. `Client` +2. `ManagedClient` + +## `Client` + +`Client` is the manual, caller-paced client. + +It owns: + +- connected UDP socket +- socket configuration +- local and remote addresses +- session state +- negotiated parameters +- connection token +- sequence state +- bounded pending-probe state + +It does not own: + +- threads +- public event hub +- statistics aggregation +- CLI formatting +- long-term sample history + +The consumer controls pacing and polling. + +### Example Shape + +```rust +pub struct Client { + socket: ClientSocket, + state: SessionState, +} + +impl Client { + pub fn connect(config: ClientConfig) -> Result; + pub fn open(&mut self, now: ClientTimestamp) -> Result, ClientError>; + pub fn next_send_deadline(&self) -> Option; + pub fn send_probe(&mut self, now: ClientTimestamp) -> Result, ClientError>; + pub fn recv_once(&mut self) -> Result>, ClientError>; + pub fn recv_available(&mut self, budget: RecvBudget) -> Result, ClientError>; + pub fn poll_timeouts(&mut self, now: ClientTimestamp) -> Result, ClientError>; + pub fn close(&mut self, now: ClientTimestamp) -> Result, ClientError>; +} +``` + +The exact API may be adjusted, but the core principle is stable: + +```text +Client owns socket/session. +Caller owns pacing. +``` + +Manual pacing is required so consumers such as `sqm-autorate-rust` can integrate probing into their own control loop. + +--- + +## `ManagedClient` + +`ManagedClient` is a convenience runner built on the same socket/session primitives. + +It owns: + +- runtime threads +- cancellation +- pacing loop +- receive loop +- coordinator/correlator +- event fanout + +It is intended for: + +- `irtt-cli` +- simple applications +- integration tests +- users who do not need manual control + +### Example Shape + +```rust +pub struct ManagedClientSession { + // owns worker threads and event hub +} + +impl ManagedClientSession { + pub fn subscribe(&self, config: SubscriberConfig) -> EventSubscription; + pub fn stop(&self); + pub fn join(self) -> Result; +} + +pub struct ManagedClient; + +impl ManagedClient { + pub fn start(config: ClientConfig) -> Result; +} +``` + +--- + +# Socket Ownership + +The IRTT session uses one connected UDP socket for open, echo, and close traffic. + +This socket is part of the client/session abstraction and should be owned by `irtt-client`, not by downstream consumers. + +Manual pacing does not imply manual socket ownership. + +## `ClientSocket` + +Use an internal socket wrapper named `ClientSocket`. + +Do not use names like `IrtSocket`. + +```rust +struct ClientSocket { + inner: std::net::UdpSocket, +} +``` + +`ClientSocket` is responsible for: + +- creating the socket +- binding local address +- configuring socket options +- connecting UDP socket +- sending packets +- receiving packets +- cloning handles if required by managed mode +- returning optional packet metadata + +## `socket2` + +Use `socket2` for socket creation and configuration. + +The MVP implementation may create/configure with `socket2::Socket`, then convert into `std::net::UdpSocket` internally. + +`socket2` is required because the standard library does not expose all socket options needed for DSCP/TOS/Traffic Class/TTL style behavior. + +## One Socket, Multiple Handles + +Managed mode may clone the underlying socket handle for sender/receiver threads. + +This is still one connected UDP socket at the OS/session level. + +--- + +# Received Packet Metadata + +ECN and DSCP are IP-header metadata. They are not part of the IRTT payload wire format. + +The client socket abstraction should expose optional packet metadata: + +```rust +pub struct PacketMeta { + pub traffic_class: Option, + pub dscp: Option, + pub ecn: Option, + pub kernel_rx_timestamp: Option, +} +``` + +## Important Modeling Rule + +The implementation must distinguish: + +- raw Traffic Class / TOS byte +- DSCP bits +- ECN bits + +It must not collapse these into a single ambiguous `ecn` field. + +## MVP + +MVP may use normal `UdpSocket::recv` / `recv_from` behavior internally. + +In this mode: + +```text +traffic_class = None +dscp = None +ecn = None +kernel_rx_timestamp = None +``` + +## Future `ancillary` Feature + +A future `ancillary` feature should use platform control-message APIs, such as `recvmsg`, to populate: + +- raw received Traffic Class / TOS byte +- derived DSCP +- derived ECN +- optional kernel receive timestamp + +Normal protocol compatibility must not depend on this feature. + +## Upstream Server ECN Mode + +Upstream server ECN mode should be treated as an advanced compatibility feature. It appears to require a control-message socket path rather than a new IRTT payload field. + +Before claiming compatibility with this mode, add dedicated black-box tests using `irtt server --ecn`. + +--- + +# DSCP / Traffic Class / ECN API + +The Rust API should expose real DSCP and ECN concepts while also permitting raw compatibility. + +```rust +pub struct Dscp(u8); // valid 0..=63 +pub struct Ecn(u8); // valid 0..=3 + +pub enum TrafficClassConfig { + Dscp(Dscp), + DscpEcn { dscp: Dscp, ecn: Ecn }, + Raw(u8), +} +``` + +CLI should distinguish: + +```text +--dscp 46 # real DSCP 46, raw byte 0xb8 +--traffic-class 0xb8 # raw byte compatibility +--ecn ect0 # optional/future explicit ECN helper +``` + +The protocol parameter ultimately negotiates/sends a raw Traffic Class / TOS byte. The public Rust API should avoid misleading users into thinking every value is a pure 6-bit DSCP codepoint. + +--- + +# Runtime Model + +## Core Decision + +Use a synchronous/threaded core for the managed runner. + +Do not make Tokio or any async runtime required by the core crates. + +Async integration may be added later as a wrapper or feature. + +## Manual `Client` + +Manual `Client` owns no threads. + +The caller decides when to: + +- open +- send probes +- poll receive traffic +- poll timeouts +- close + +This is the preferred integration layer for `sqm-autorate-rust` if it wants direct control. + +## Managed Runtime Topology + +After negotiation, `ManagedClient` should use three runtime roles: + +1. coordinator/control thread +2. sender/pacer thread +3. receiver thread + +### Sender/Pacer Thread + +The sender/pacer thread is timing-sensitive. + +It must not perform: + +- statistics aggregation +- public event fanout +- CLI formatting +- subscriber delivery +- long-running coordination work +- blocking operations unrelated to sending + +It should only: + +- maintain absolute send schedule +- prepare outgoing packets before deadline where possible +- send packets +- timestamp successful sends immediately after send returns +- report compact sent records to coordinator + +### Receiver Thread + +The receiver thread should: + +- block on receive +- timestamp immediately after receive returns, before expensive parsing/validation +- parse/validate reply enough to create a compact reply record +- report compact reply records to coordinator + +### Coordinator Thread + +The coordinator should: + +- correlate sent and received records +- compute RTT samples +- manage pending probes +- handle loss timeouts +- detect duplicates/late replies +- emit public events through the event hub +- handle graceful shutdown and close + +--- + +# Event Distribution + +`flume` channels are single-consumer queues, not broadcast channels. + +Therefore `ManagedClient` must not expose a single shared `Receiver` as the only event API. + +Use `flume` internally, but provide a small fanout/broadcast layer for public subscribers. + +## Event Hub + +```rust +pub struct EventHub { + // internal list of subscriber queues +} + +pub struct EventSubscription { + receiver: flume::Receiver, + id: SubscriberId, +} +``` + +Consumers subscribe independently: + +```rust +let stats_rx = session.subscribe(SubscriberConfig::stats()); +let sqm_rx = session.subscribe(SubscriberConfig::latest_only()); +let cli_rx = session.subscribe(SubscriberConfig::cli()); +``` + +Each subscriber receives its own copy of each event. + +## Subscriber Configuration + +```rust +pub struct SubscriberConfig { + pub capacity: usize, + pub overflow: SubscriberOverflow, + pub event_filter: EventFilter, +} + +pub enum SubscriberOverflow { + DropNewest, + DropOldest, + Block, + Disconnect, +} +``` + +Suggested profiles: + +```text +live control / sqm: + DropOldest, small bounded queue, keep freshest data + +stats: + Block or Disconnect, because dropped events corrupt statistics + +debug logging: + DropNewest +``` + +The event hub should track dropped events per subscriber where practical. + +--- + +# Public Events + +Events should be compact and cheap to clone. + +Do not include large packet buffers in normal public events. + +## Core Events + +```rust +pub enum ClientEvent { + SessionStarted { + session_id: SessionId, + remote: SocketAddr, + negotiated: NegotiatedParams, + }, + + EchoReply { + session_id: SessionId, + seq: u64, + wire_seq: u32, + remote: SocketAddr, + sent_at: ClientTimestamp, + received_at: ClientTimestamp, + rtt: RttSample, + one_way: Option, + server_timing: Option, + received_stats: Option, + packet_meta: PacketMeta, + }, + + EchoLoss { + session_id: SessionId, + seq: u64, + wire_seq: u32, + kind: LossKind, + }, + + DuplicateReply { + session_id: SessionId, + seq: u64, + wire_seq: u32, + }, + + LateReply { + session_id: SessionId, + seq: u64, + wire_seq: u32, + rtt: Option, + }, + + Warning { + session_id: Option, + warning: ClientWarning, + }, + + SessionEnded { + session_id: SessionId, + reason: SessionEndReason, + }, + + FatalError { + session_id: Option, + error: ClientErrorKind, + }, +} +``` + +`EchoSent` should not be emitted by default. It may be available as an optional debug event. + +--- + +# Sequence Numbers + +The wire protocol sequence number is 32-bit. + +Public events should expose a logical `u64` sequence number and optionally also the wire `u32` sequence. + +```rust +struct ProbeSeq { + logical: u64, + wire: u32, +} +``` + +This keeps continuous mode sane even if wire sequence wraps in very long sessions. + +--- + +# Run Modes + +```rust +pub enum RunMode { + NoTest, + Duration(Duration), + Count(u64), + Continuous, +} +``` + +## `NoTest` + +Open and immediately close/complete without echo probes. + +Used for: + +- CLI check mode +- negotiation smoke tests +- interop tests + +## `Duration` + +Finite duration-limited test. + +Finite duration has an exclusive end. Expected packet count for finite duration is: + +```text +ceil(duration / interval) +``` + +## `Count` + +Finite packet-count test. + +Useful for deterministic tests and simple manual checks. + +## `Continuous` + +Continuous mode is first-class. + +It must: + +- emit events until stopped +- avoid full sample retention +- avoid preallocating finite result vectors +- keep memory bounded +- send close packet on graceful stop where possible + +If the server permits unbounded or sufficiently long sessions, use a single long-lived session. + +If the server negotiates a finite duration or closes the session, behavior is controlled by `ReconnectPolicy`. + +```rust +pub enum ReconnectPolicy { + Never, + OnServerClose { delay: Duration }, + Always { delay: Duration, max_attempts: Option }, +} +``` + +Suggested defaults: + +```text +library default: + ReconnectPolicy::Never + +CLI continuous default: + ReconnectPolicy::OnServerClose { delay: 1s } +``` + +--- + +# Negotiation Policy + +Do not use a confusing strict/loose boolean as the primary API. + +Use: + +```rust +pub enum NegotiationPolicy { + AcceptServerRestrictions, + RequireExact, + RequireWithin(LocalRequirements), +} +``` + +Suggested defaults: + +```text +irtt-client default: + AcceptServerRestrictions + +interop tests: + RequireExact where useful + +sqm/cake usage: + RequireWithin(...) when the caller has hard operational limits +``` + +The server is authoritative unless the caller explicitly requires exact or bounded negotiated values. + +--- + +# Timing Policy + +## Send Timestamp + +Capture send timestamp immediately after `send` returns. + +## Receive Timestamp + +Capture receive timestamp immediately after receive returns, before expensive validation/parsing. + +If kernel receive timestamps are available through an optional ancillary feature, expose them separately in `PacketMeta` or timestamp metadata. + +## Timer Scheduling + +Managed sender uses absolute scheduling: + +```text +scheduled_time(seq) = session_start + seq * interval +``` + +Do not schedule by repeatedly doing: + +```text +next = now + interval +``` + +because that accumulates drift. + +## Timer Modes + +```rust +pub enum TimerMode { + Normal, + Precise { spin_threshold: Duration }, +} +``` + +Default: + +```text +TimerMode::Normal +``` + +Router mode must not busy-spin unless explicitly requested. + +--- + +# RTT Model + +Do not collapse RTT into a single ambiguous value. + +Expose raw RTT, server processing time, and adjusted RTT separately. + +```rust +pub struct RttSample { + pub raw: Duration, + pub adjusted: Option, + pub server_processing: Option, +} + +impl RttSample { + pub fn effective(&self) -> Duration { + self.adjusted.unwrap_or(self.raw) + } +} +``` + +## Rules + +```text +raw RTT: + client_receive_mono - client_send_mono + +server processing: + server_send - server_receive, if both relevant server timestamps are available + +adjusted RTT: + raw - server_processing, only if server_processing <= raw + +if server_processing > raw: + adjusted = None + emit warning + effective RTT falls back to raw +``` + +Warning type: + +```rust +pub enum ClientWarning { + ServerProcessingExceedsRawRtt { + seq: u64, + raw: Duration, + server_processing: Duration, + }, +} +``` + +Consumers decide whether to use raw, adjusted, or effective RTT. + +Suggested defaults: + +```text +sqm-autorate-rust: + use rtt.effective() + +simple CLI/rtt-us format: + print rtt.effective() + +machine CLI format: + include effective, raw, and server-processing fields + +stats: + provide effective by default, but expose raw/adjusted variants where useful +``` + +--- + +# Loss Model + +`irtt-client` may emit loss events based on timeout/loss policy. + +```rust +pub enum LossDetection { + Disabled, + Timeout(Duration), + MultipleOfInterval(u32), +} +``` + +Suggested default for continuous mode: + +```text +LossDetection::MultipleOfInterval(3) +``` + +Loss kind: + +```rust +pub enum LossKind { + Unknown, + Upstream, + Downstream, +} +``` + +Direction classification should use received count/window when available. If insufficient data exists, use `Unknown`. + +--- + +# Validation Policy + +```rust +pub enum ValidationPolicy { + Strict, + DropMalformed, + DropUnrelatedAbortMalformedSession, +} +``` + +Suggested default: + +```text +Drop unrelated garbage. +Abort or fatal-error on packets that appear to belong to the active session but violate core protocol invariants. +``` + +Examples: + +```text +wrong magic / unrelated packet: + drop + +bad HMAC / bad token / impossible sequence / malformed active-session packet: + abort or emit FatalError depending on policy +``` + +--- + +# Cancellation and Outcome + +No signal handling in `irtt-client`. + +`irtt-cli` handles Ctrl-C and calls `stop()` / `join()`. + +```rust +pub struct CancellationToken { + // Arc internally +} + +pub struct SessionOutcome { + pub end_reason: SessionEndReason, + pub packets_sent: u64, + pub replies_received: u64, + pub duplicates: u64, + pub late: u64, + pub malformed: u64, +} +``` + +For manual `Client`, the caller owns cancellation by simply not calling further send/poll methods and then calling close/drop as appropriate. + +--- + +# CLI Output + +CLI output must support router-friendly, awk-friendly, line-oriented formats. + +The design is inspired by classic ping-style machine output: + +- one result/event per line +- fixed positional fields +- configurable delimiter +- optional timestamp +- separate human and machine output +- no JSON parser required for router usage + +## Formats + +Support: + +```text +human +machine +simple +rtt-us +jsonl # optional feature +``` + +## Machine Format + +Default delimiter: tab. + +Reply line: + +```text +replytimestampremotesession_idseqrtt_usraw_rtt_usserver_processing_usdown_usup_usecnstatus +``` + +Loss line: + +```text +losstimestampremotesession_idseqloss_kind +``` + +Duplicate line: + +```text +duplicatetimestampremotesession_idseq +``` + +Late line: + +```text +latetimestampremotesession_idseqrtt_us +``` + +Session line: + +```text +sessiontimestampremotesession_idstatereason +``` + +Example: + +```text +reply 1714320000.123456 1.2.3.4:2112 1 42 8341 8420 79 - - - ok +loss 1714320000.323456 1.2.3.4:2112 1 43 unknown +``` + +Field meanings: + +```text +rtt_us: + effective RTT = adjusted if valid, else raw + +raw_rtt_us: + raw client-measured RTT + +server_processing_us: + server processing time or "-" + +down_us/up_us: + one-way estimates if available, otherwise "-" + +ecn: + received ECN if available, otherwise "-" +``` + +## Simple Format + +```text +timestampseqresult +``` + +Example: + +```text +1714320000.123456 42 8341 +1714320000.323456 43 loss +1714320000.523456 44 8120 +``` + +## RTT-Only Format + +```text +8341 +loss +8120 +``` + +This is intended for extremely simple shell consumption. + +## Delimiter + +Support: + +```text +--delimiter tab +--delimiter ',' +--delimiter ';' +--delimiter '|' +``` + +Internally, join fields. Do not mutate format strings. + +## Timestamp Format + +Default timestamp format: + +```text +unix_seconds.microseconds +``` + +Example: + +```text +1714320000.123456 +``` + +Avoid nanosecond epoch integers in awk-oriented output. + +## Header + +Default: no header. + +Optional: + +```text +--header +``` + +Machine-readable output formats must be stable. Fields must not be reordered between releases. New fields should be added only in a new named/versioned format or at the end when explicitly documented. + +--- + +# Dependencies + +## `irtt-proto` + +```toml +[dependencies] +thiserror = "2" +hmac = "0.12" +md-5 = "0.10" +subtle = "2" +fastrand = "2" +serde = { version = "1", features = ["derive"], optional = true } +``` + +## `irtt-client` + +```toml +[dependencies] +irtt-proto = { path = "../irtt-proto" } +thiserror = "2" +socket2 = "0.5" +flume = "0.11" +fastrand = "2" +serde = { version = "1", features = ["derive"], optional = true } +tracing = { version = "0.1", optional = true } +``` + +Future optional dependency for ancillary receive metadata: + +```toml +rustix = { version = "...", optional = true, features = ["net"] } +``` + +Exact ancillary dependency can be chosen later. + +## `irtt-stats` + +```toml +[dependencies] +irtt-client = { path = "../irtt-client" } +thiserror = "2" +serde = { version = "1", features = ["derive"], optional = true } +``` + +## `irtt-cli` + +```toml +[dependencies] +irtt-client = { path = "../irtt-client" } +irtt-stats = { path = "../irtt-stats", optional = true } +clap = { version = "4", features = ["derive"] } +thiserror = "2" +ctrlc = "3" +serde_json = { version = "1", optional = true } +``` + +--- + +# Testing Strategy + +## Pure Tests + +Normal `cargo test` should run: + +- varint encode/decode tests +- parameter encode/decode tests +- packet layout calculation tests +- open request encode tests +- open reply decode tests +- echo request encode tests +- echo reply decode tests +- close request encode tests +- HMAC calculation and verification tests +- clean packet vector tests +- stats unit tests +- CLI formatter unit tests + +## Interop Tests + +Interop tests require `irtt` in `PATH`. + +Local default: + +```bash +cargo test +``` + +Interop: + +```bash +cargo test --workspace --features interop +``` + +CI should run interop tests and provide upstream `irtt` in `PATH`. + +Local development should not require `irtt` unless the interop feature is explicitly enabled. + +## `tshark` + +`tshark` is diagnostic/manual only. + +Do not make `tshark` a normal test or CI dependency. + +## Black-Box ECN Tests + +Before claiming support for upstream server ECN mode, run dedicated black-box tests for: + +- IPv4 behavior +- IPv6 behavior +- server `--ecn` +- outgoing Traffic Class values with ECN bits set +- received Traffic Class metadata +- logged ECN values +- behavior with and without ancillary receive support + +--- + +# Implementation Milestones + +## Milestone 0 — Workspace Skeleton + +- create Cargo workspace +- create crates +- add license files +- add clean-room docs +- add design docs +- ensure no upstream source is present + +## Milestone 1 — `irtt-proto` + +- flags +- constants +- varint/zigzag +- params +- packet layout +- open request/reply +- echo request/reply +- close request +- HMAC +- packet vectors passing + +## Milestone 2 — `irtt-client::Client` No-Test / Negotiation + +- `ClientSocket` with `socket2` +- connected UDP socket +- open request +- open reply parse +- no-test mode +- close behavior +- basic interop with upstream server + +## Milestone 3 — Manual Client Finite Probe Events + +- caller-paced send_probe +- recv_once / recv_available +- timeout polling +- EchoReply events +- RTT sample model +- duplicate/late detection +- finite count/duration smoke test + +## Milestone 4 — Minimal `irtt-cli` Stream Mode + +- use manual or managed layer as appropriate +- machine/simple/rtt-us output +- delimiter support +- timestamp support +- no stats required for minimal build + +## Milestone 5 — Managed Runner + +- coordinator/control thread +- sender/pacer thread +- receiver thread +- EventHub fanout +- cancellation +- stop/join +- continuous mode + +## Milestone 6 — `irtt-stats` + +- consumes `ClientEvent` +- cumulative stats +- rolling stats +- finite summary +- min/max/mean/median/stddev/variance +- loss/duplicate/late summaries + +## Milestone 7 — Full/default CLI Stats Integration + +- default build enables stats +- finite human report +- optional stats lines in stream mode +- minimal `--no-default-features` build still works without stats + +## Milestone 8 — Compatibility Features + +- HMAC interop +- all timestamp modes +- all received stats modes +- server fill +- DSCP / Traffic Class +- TTL / hop limit where applicable + +## Milestone 9 — Advanced Networking + +- ancillary receive metadata feature +- incoming Traffic Class / DSCP / ECN +- optional kernel RX timestamps +- upstream server `--ecn` compatibility tests + +--- + +# Known Open / Verification Items + +These are not blockers for the initial architecture, but should be tracked: + +- exact behavior of upstream server ECN mode +- IPv4 vs IPv6 ECN metadata behavior +- kernel RX timestamp portability +- server close during active test +- rare case where server processing time exceeds raw RTT +- exact handling of very long or unbounded continuous sessions +- ancillary implementation strategy on Linux/OpenWrt/macOS + +Design should be defensive around these items and expose data to consumers rather than hiding ambiguity. + +--- + +# Implementation Agent Rules + +The implementation agent must follow these rules: + +1. Do not inspect upstream source code. +2. Do not inspect upstream tests. +3. Do not inspect contaminated notes or transcripts. +4. Implement from the clean protocol spec, black-box report, packet vectors, and this Rust design brief. +5. Do not build an upstream CLI clone first. +6. Keep `irtt-client` event-first and stats-free. +7. Keep stats in `irtt-stats`. +8. Keep CLI formatting in `irtt-cli`. +9. Keep ECN/DSCP receive metadata out of `irtt-proto`. +10. Keep managed runtime optional; expose manual `Client` for caller-paced integration. + +Final architecture mantra: + +```text +irtt-proto: wire protocol +irtt-client::Client: socket + session, caller-paced +irtt-client::ManagedClient: threaded convenience runner +irtt-stats: event aggregation +irtt-cli: user-facing formatting and reports +``` + diff --git a/docs/test-vectors.md b/docs/test-vectors.md new file mode 100644 index 0000000..d9f2cbc --- /dev/null +++ b/docs/test-vectors.md @@ -0,0 +1,304 @@ +# IRTT Packet Test Vectors + +**Source:** Black-box captures from irtt 0.9.1 on macOS Darwin 25.3.0 arm64 +**Date:** 2026-04-28 + +All vectors are from actual network captures (not constructed). + +--- + +## Vector 1: Open Request (no HMAC) + +**Scenario:** Client opens session with default parameters (duration=3s, +interval=1s, stats=both, tstamp=both, clock=both). +**Direction:** Client → Server +**irtt version:** 0.9.1 +**Command:** `irtt client -d 3s -i 1s 127.0.0.1:2112` + +**Packet (24 bytes):** +``` +14 a7 5b 01 01 02 02 80 f8 82 ad 16 03 80 a8 d6 +b9 07 05 06 06 06 07 06 +``` + +**Field breakdown:** +- Bytes 0-2: Magic `14 a7 5b` +- Byte 3: Flags `01` (Open) +- Bytes 4-23: Parameter payload: + - Tag 1 (ProtocolVersion): value `02` → zigzag → 1 + - Tag 2 (Duration): value `80 f8 82 ad 16` → zigzag → 3000000000 ns (3s) + - Tag 3 (Interval): value `80 a8 d6 b9 07` → zigzag → 1000000000 ns (1s) + - Tag 5 (ReceivedStats): value `06` → zigzag → 3 (Both) + - Tag 6 (StampAt): value `06` → zigzag → 3 (Both) + - Tag 7 (Clock): value `06` → zigzag → 3 (Both) + +**Notes:** +- No connection token field (token not yet assigned) +- Tags 4 (Length), 8 (DSCP) omitted because values are 0 +- Tag 9 (ServerFill) omitted because not specified + +--- + +## Vector 2: Open Reply (no HMAC) + +**Scenario:** Server accepts session, returns token and echoed parameters. +**Direction:** Server → Client +**Token:** `0x7896b6ab87715213` + +**Packet (32 bytes):** +``` +14 a7 5b 03 13 52 71 87 ab b6 96 78 01 02 02 80 +f8 82 ad 16 03 80 a8 d6 b9 07 05 06 06 06 07 06 +``` + +**Field breakdown:** +- Bytes 0-2: Magic `14 a7 5b` +- Byte 3: Flags `03` (Open | Reply) +- Bytes 4-11: Connection token `13 52 71 87 ab b6 96 78` (LE: 0x7896b6ab87715213) +- Bytes 12-31: Parameter payload (same as request — no restrictions) + +--- + +## Vector 3: Echo Request (no HMAC, default stats+timestamps) + +**Scenario:** First test packet, sequence 0, default settings. +**Direction:** Client → Server +**Token:** `0x7896b6ab87715213`, Seq: 0 + +**Packet (60 bytes):** +``` +14 a7 5b 00 13 52 71 87 ab b6 96 78 00 00 00 00 +00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 +00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 +00 00 00 00 00 00 00 00 00 00 00 00 +``` + +**Field breakdown:** +- Bytes 0-2: Magic `14 a7 5b` +- Byte 3: Flags `00` (no flags — echo request from client) +- Bytes 4-11: Connection token +- Bytes 12-15: Sequence number `00 00 00 00` (LE: 0) +- Bytes 16-19: Received count (zeroed placeholder) +- Bytes 20-27: Received window (zeroed placeholder) +- Bytes 28-35: Receive wall timestamp (zeroed placeholder) +- Bytes 36-43: Receive mono timestamp (zeroed placeholder) +- Bytes 44-51: Send wall timestamp (zeroed placeholder) +- Bytes 52-59: Send mono timestamp (zeroed placeholder) + +**Notes:** All stats and timestamp fields are zeroed in echo requests. +They serve as placeholders to reach the negotiated packet length. + +--- + +## Vector 4: Echo Reply (no HMAC, default stats+timestamps) + +**Scenario:** Server reply to seq 2, third packet in session. +**Direction:** Server → Client +**Token:** `0x7896b6ab87715213`, Seq: 2, Recv Count: 3, Window: 0x07 + +**Packet (60 bytes):** +``` +14 a7 5b 02 13 52 71 87 ab b6 96 78 02 00 00 00 +03 00 00 00 07 00 00 00 00 00 00 00 b8 1a 33 0c +86 6d aa 18 de 26 35 95 00 00 00 00 80 4d 33 0c +86 6d aa 18 b2 57 35 95 00 00 00 00 +``` + +**Field breakdown:** +- Bytes 0-2: Magic `14 a7 5b` +- Byte 3: Flags `02` (Reply) +- Bytes 4-11: Connection token +- Bytes 12-15: Sequence number `02 00 00 00` (LE: 2) +- Bytes 16-19: Received count `03 00 00 00` (LE: 3) +- Bytes 20-27: Received window `07 00 00 00 00 00 00 00` (LE: 0x07, bits 0-2 set) +- Bytes 28-35: Receive wall timestamp `b8 1a 33 0c 86 6d aa 18` (LE int64) +- Bytes 36-43: Receive mono timestamp `de 26 35 95 00 00 00 00` (LE int64) +- Bytes 44-51: Send wall timestamp `80 4d 33 0c 86 6d aa 18` (LE int64) +- Bytes 52-59: Send mono timestamp `b2 57 35 95 00 00 00 00` (LE int64) + +**Window interpretation (for seq 2):** +- Bit 0 (0x01): Seq 2 received ✓ +- Bit 1 (0x02): Seq 1 received ✓ +- Bit 2 (0x04): Seq 0 received ✓ + +--- + +## Vector 5: Close Request (no HMAC) + +**Scenario:** Client sends close after test. +**Direction:** Client → Server +**Token:** `0x7896b6ab87715213` + +**Packet (12 bytes):** +``` +14 a7 5b 04 13 52 71 87 ab b6 96 78 +``` + +**Field breakdown:** +- Bytes 0-2: Magic `14 a7 5b` +- Byte 3: Flags `04` (Close) +- Bytes 4-11: Connection token + +--- + +## Vector 6: HMAC Open Request + +**Scenario:** Client opens session with HMAC key "testkey". +**Direction:** Client → Server +**HMAC key:** ASCII "testkey" (7 bytes) + +**Packet (40 bytes):** +``` +14 a7 5b 09 ff 90 16 a7 aa 53 78 16 9e c3 a2 d5 +54 dc 30 36 01 02 02 80 d0 ac f3 0e 03 80 a8 d6 +b9 07 05 06 06 06 07 06 +``` + +**Field breakdown:** +- Bytes 0-2: Magic `14 a7 5b` +- Byte 3: Flags `09` (Open | HMAC) +- Bytes 4-19: HMAC `ff 90 16 a7 aa 53 78 16 9e c3 a2 d5 54 dc 30 36` +- Bytes 20-39: Parameter payload + +**HMAC verified independently with Python (HMAC-MD5 over entire packet +with HMAC field zeroed).** + +--- + +## Vector 7: HMAC Echo Request + +**Scenario:** Echo request with HMAC, seq 0. +**Direction:** Client → Server +**Token:** `0x4387e9eb5d3fca59`, HMAC key: "testkey" + +**Packet (76 bytes):** +``` +14 a7 5b 08 d9 87 4f 82 b3 13 10 31 59 ad 2c 8f +6b da ef ff 59 ca 3f 5d eb e9 87 43 00 00 00 00 +00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 +00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 +00 00 00 00 00 00 00 00 00 00 00 00 +``` + +**Field breakdown:** +- Bytes 0-2: Magic +- Byte 3: Flags `08` (HMAC) +- Bytes 4-19: HMAC +- Bytes 20-27: Connection token `59 ca 3f 5d eb e9 87 43` +- Bytes 28-31: Sequence number `00 00 00 00` +- Bytes 32-75: Stats + timestamp placeholders (zeroed) + +--- + +## Vector 8: HMAC Close Request + +**Scenario:** Close with HMAC. +**Direction:** Client → Server +**Token:** `0x4387e9eb5d3fca59`, HMAC key: "testkey" + +**Packet (28 bytes):** +``` +14 a7 5b 0c f5 cd 0f a9 de 9d 7d 66 8d 91 a5 32 +48 0e 42 e0 59 ca 3f 5d eb e9 87 43 +``` + +**Field breakdown:** +- Bytes 0-2: Magic +- Byte 3: Flags `0c` (Close | HMAC) +- Bytes 4-19: HMAC +- Bytes 20-27: Connection token + +--- + +## Vector 9: No-Test Mode (Open+Close Request) + +**Scenario:** Client opens and immediately closes. +**Direction:** Client → Server + +**Packet (25 bytes):** +``` +14 a7 5b 05 01 02 02 80 e0 ba 84 bf 03 03 80 a8 +d6 b9 07 05 06 06 06 07 06 +``` + +**Field breakdown:** +- Bytes 0-2: Magic +- Byte 3: Flags `05` (Open | Close) +- Bytes 4-24: Parameter payload + - Tag 1: ProtocolVersion = 1 + - Tag 2: Duration = 60000000000 ns (60s, default) + - Tag 3: Interval = 1000000000 ns (1s, default) + - Tag 5: ReceivedStats = 3 (Both) + - Tag 6: StampAt = 3 (Both) + - Tag 7: Clock = 3 (Both) + +--- + +## Vector 10: No-Test Mode (Open+Close Reply) + +**Scenario:** Server acknowledges no-test open+close. +**Direction:** Server → Client + +**Packet (33 bytes):** +``` +14 a7 5b 07 00 00 00 00 00 00 00 00 01 02 02 80 +e0 ba 84 bf 03 03 80 a8 d6 b9 07 05 06 06 06 07 +06 +``` + +**Field breakdown:** +- Bytes 0-2: Magic +- Byte 3: Flags `07` (Open | Reply | Close) +- Bytes 4-11: Connection token `00 00 00 00 00 00 00 00` (ZERO) +- Bytes 12-32: Parameter payload + +**Notes:** In no-test mode, the server returns a zero connection token. +The Close flag in this context means "acknowledged close" not "rejected." + +--- + +## Vector 11: Minimal Echo Packet (no stats, no timestamps) + +**Scenario:** Echo request with stats=none, tstamp=none. +**Direction:** Client → Server +**Token:** `0xa0316fa25c61154e`, Seq: 0 + +**Packet (16 bytes):** +``` +14 a7 5b 00 4e 15 61 5c a2 6f 31 a0 00 00 00 00 +``` + +**Field breakdown:** +- Bytes 0-2: Magic +- Byte 3: Flags `00` +- Bytes 4-11: Connection token +- Bytes 12-15: Sequence number + +This is the absolute minimum echo packet size. + +--- + +## Vector 12: Midpoint Timestamps (both clocks) + +**Scenario:** Echo reply with stats=both, tstamp=midpoint, clock=both. +**Direction:** Server → Client + +**Packet (44 bytes):** +``` +14 a7 5b 02 91 a4 1e 7a f0 14 f6 62 00 00 00 00 +01 00 00 00 01 00 00 00 00 00 00 00 f7 0e 2c 93 +a2 6d aa 18 5a 5f 14 47 03 00 00 00 +``` + +**Field breakdown:** +- Bytes 0-2: Magic +- Byte 3: Flags `02` (Reply) +- Bytes 4-11: Token +- Bytes 12-15: Seq `00 00 00 00` (0) +- Bytes 16-19: Recv count `01 00 00 00` (1) +- Bytes 20-27: Recv window `01 00 00 00 00 00 00 00` +- Bytes 28-35: Midpoint wall timestamp +- Bytes 36-43: Midpoint mono timestamp + +**Notes:** Only midpoint timestamps present. No receive/send timestamp +fields. Confirms midpoint is mutually exclusive with receive+send.