320 lines
19 KiB
Markdown
320 lines
19 KiB
Markdown
# All-target matrix validation — 2026-10-13
|
|
|
|
The Gitea workflow now defaults to `targets=all`. Target discovery reads every
|
|
page of the public GHCR SDK tag list and selects each release's target tags,
|
|
using release directory names to exclude architecture aliases. Architecture
|
|
metadata comes from the release's `.targets.json`, with the matching OpenWrt
|
|
source tag supplying `ARCH` for SDK-only targets absent from that JSON.
|
|
|
|
Live generation and registry manifest checks produced:
|
|
|
|
| OpenWrt release | Available target SDK images | Rust-supported matrix targets |
|
|
| --- | ---: | ---: |
|
|
| 25.12.5 (stable) | 94 | 93 |
|
|
| 24.10.8 (oldstable) | 90 | 89 |
|
|
| Total | 184 | 182 |
|
|
|
|
Both releases exclude `ixp4xx/generic` (`armeb`). The architecture allowlist
|
|
matches `RUST_ARCH_DEPENDS` at the released packages feed revisions:
|
|
[25.12.5](https://github.com/openwrt/packages/blob/5caa62e0bc9f7fb9b0c12a23267bceb7724214dd/lang/rust/rust-values.mk)
|
|
and [24.10.8](https://github.com/openwrt/packages/blob/23abaa6f3b0fdfd76b570031107e5718476ff0c8/lang/rust/rust-values.mk).
|
|
All four Malta subtargets remain included for both releases. The older release
|
|
also retains SDK-only `lantiq/ase`, `lantiq/xway_legacy`, `ramips/rt288x` and
|
|
`realtek/rtl931x`.
|
|
|
|
Validation passed:
|
|
|
|
- All 182 generated base-image tags returned HTTP 200 and a manifest digest
|
|
from GHCR. Matrix entries are unique; `GITHUB_OUTPUT` matches JSON stdout.
|
|
- The explicit `x86/64,mediatek/filogic,octeon/generic` subset produces six
|
|
jobs. Selecting `siflower/sf21` produces only its 25.12.5 job.
|
|
- Six offline regression tests cover per-release availability, unsupported
|
|
architectures, explicit selections and duplicates, invalid/empty selections,
|
|
architecture aliases, SDK-only architecture fallback, missing metadata and
|
|
registry pagination. The matrix job runs these tests before live discovery.
|
|
- YAML parsing, actionlint 1.7.12 (excluding only the existing Gitea
|
|
`builtin:checkout` syntax), ShellCheck and `git diff --check` pass.
|
|
- Workflow assertions confirm `fail-fast: false`, unchanged `max-parallel: 1`,
|
|
timeout, runner, build/verification commands and cache environment. Each job
|
|
pushes after its own successful verification when `publish=true`.
|
|
- Mock Docker execution checks publication and cleanup after a successful push,
|
|
a failed push and an absent image. Cleanup runs with `always()` and removes
|
|
only the job's SDK tag; it does not prune caches, volumes or other images.
|
|
|
|
The Redis/S3 infrastructure, sccache configuration, eight-job compilation limit,
|
|
eight-CPU/20 GiB builder limits and 50 GiB BuildKit GC budget are unchanged.
|
|
This validation did not compile all SDKs or dispatch/publish images. Architecture
|
|
eligibility and available base images do not guarantee every Rust bootstrap will
|
|
succeed. Live workflow/publication acceptance remains pending; the legacy
|
|
Jenkinsfile is retained unchanged until cutover acceptance.
|
|
|
|
# Migration validation — 2026-10-10
|
|
|
|
Inspected the original Jenkinsfile, Dockerfile, downstream `sqm-autorate-rust`
|
|
workflows and package Makefile, Terraform runner definition, and live TheBeast
|
|
runner configuration before changing the repository.
|
|
|
|
## Passed
|
|
|
|
- YAML parsing; TOML parsing; `sh -n`; ShellCheck; `git diff --check`.
|
|
- actionlint 1.7.12, excluding only its unsupported `builtin:checkout` action
|
|
syntax. The workflow uses the existing `ubuntu-latest` label. Gitea's documentation and the
|
|
existing downstream workflow confirm built-in checkout support.
|
|
- Default matrix resolves six entries for stable 25.12.5 and oldstable 24.10.8.
|
|
`targets=all` ran upstream target discovery inside the actual job image and
|
|
resolved 194 entries. Empty/invalid targets fail. Every default base-image
|
|
tag was read successfully from GHCR.
|
|
- Docker Compose configuration accepted; Redis healthy with its named volume,
|
|
AOF enabled and successful AOF writes. Service and bounded builder were
|
|
provisioned on TheBeast without changing the runner configuration.
|
|
- Builder inspected: eight-CPU quota, 20 GiB memory/swap limit, one BuildKit
|
|
worker step, 50 GiB layer GC budget. A second disposable job container with
|
|
fresh Docker client configuration reused the existing daemon and builder.
|
|
- Dockerfile build checks completed. Two expected advisory warnings: mandatory
|
|
base-image arguments have no defaults, and Docker's secret-name heuristic
|
|
mistakes the public `SCCACHE_REDIS_KEY_PREFIX` for a secret.
|
|
- `configured` Dockerfile stage built for x86/64 on both OpenWrt releases.
|
|
Pinned feeds select Rust 1.94.0 on 25.12.5 and 1.90.0 on 24.10.8. Both retain
|
|
`CONFIG_RUST_SCCACHE=y` through defconfig and contain the explicit bootstrap
|
|
`build.ccache=sccache` setting with LLVM CI downloads disabled.
|
|
- Actual OpenWrt make-variable expansion injects sccache into host and package
|
|
compilation, despite unexporting an inherited wrapper.
|
|
- Real Cargo library compilation in Docker BuildKit: one Redis miss. A fresh
|
|
runtime container using the Docker-network endpoint: one hit, zero misses.
|
|
- Recreated the newly provisioned Redis container using the same named volume;
|
|
a fresh compiler container still recorded one hit, zero misses. No existing
|
|
infrastructure containers were restarted.
|
|
- Same source/cache namespace with GNU versus musl target: first musl build
|
|
missed; repetition hit. Rust 1.90.0 versus 1.91.0: first 1.91 build missed;
|
|
repetition hit. All probes reported zero cache read/write errors.
|
|
- A disposable OpenWrt package used the real `Build/Compile/Cargo` helper,
|
|
SDK GCC/musl linker, OpenWrt LTO/optimization flags and a local Rust library
|
|
dependency. It built successfully in two separate containers: one dependency
|
|
miss, then one hit. The linked executable correctly remained non-cacheable.
|
|
This used a prebuilt Rust 1.90 toolchain for the probe, not a freshly compiled
|
|
SDK toolchain; it is not a full SDK/source-bootstrap acceptance result.
|
|
- Retained `ci/verify-cache.sh` passed its separate-container miss/hit checks
|
|
against that prebuilt validation toolchain. Its original hardcoded SDK rustc
|
|
path was later corrected after the live mediatek/filogic build exposed the
|
|
target-specific staging layout; see the follow-up below.
|
|
- Publication configuration checked: original Docker Hub namespace/tag format,
|
|
verification before push, explicit publish input, temporary credential
|
|
directory, no automatic publication trigger. No registry login or push was
|
|
attempted.
|
|
|
|
## Remaining acceptance
|
|
|
|
1. Enable repository Actions and configure the two Docker Hub secrets. No
|
|
runner configuration changes are required.
|
|
2. Complete a non-publishing Gitea dispatch for the full default matrix.
|
|
The mediatek/filogic 24.10.8 job completed Rust source bootstrap and image
|
|
export; its corrected verification passed on the built image. Acceptance
|
|
for the other combinations remains pending.
|
|
3. With explicit publication approval, run the publishing dispatch, pull the
|
|
resulting tags and build the downstream OpenWrt package with the new image.
|
|
Actual Docker Hub authorization and publication remain unverified.
|
|
4. After acceptance, disable the old Jenkins job and remove `Jenkinsfile`.
|
|
It is deliberately retained until the replacement passes these checks.
|
|
|
|
Remote test logs and disposable probe sources are under
|
|
`/tmp/openwrt-sdk-rust-migration` in TheBeast's `gitea-runner` instance. The
|
|
Redis volume/service and BuildKit builder remain available for the replacement
|
|
pipeline; no cache volume or existing infrastructure was deleted.
|
|
|
|
## Follow-up: mediatek/filogic 24.10.8 verification
|
|
|
|
The supplied job 1602 log shows full SDK compilation and image export completed
|
|
successfully. Verification failed with exit 127 because it assumed
|
|
`staging_dir/host/bin/rustc`. Inspection of that exact built image reproduced
|
|
the failure and located Rust/Cargo in
|
|
`staging_dir/target-aarch64_cortex-a53_musl/host/bin` instead. Both the workflow
|
|
and retained cache probe now resolve the unique executable compiler from the
|
|
SDK staging tree, rather than assuming a shared host directory.
|
|
|
|
Reran the corrected verification on that image: Rust 1.90.0 and Cargo 1.90.0
|
|
execute successfully. The retained cache probe reports one Redis miss in its
|
|
first container and one hit in a fresh second container, with zero cache
|
|
read/write errors. ShellCheck, actionlint (excluding only Gitea's built-in
|
|
checkout syntax), and `git diff --check` pass. No rebuild or publication was
|
|
performed for this fix.
|
|
|
|
## Follow-up: mediatek/filogic 25.12.5 bootstrap concurrency
|
|
|
|
Job 1650 failed inside LLVM compilation with
|
|
`c++: fatal error: Killed signal terminated program cc1plus`. Its diagnostic
|
|
Make retry still configured `LLVM_PARALLEL_COMPILE_JOBS=32`. The persistent
|
|
builder's cgroup reports three OOM kills and its memory limit remains 20 GiB;
|
|
the counter is cumulative, while the supplied log identifies the killed
|
|
compiler in this job. Outer Make parallelism had not bounded Rust bootstrap's
|
|
independent job count.
|
|
|
|
The SDK Rust Makefile now passes `RUST_BUILD_JOBS` to `x.py --jobs`, defaulting
|
|
to eight for subsequent SDK use. `build-rust` supplies eight (or the requested
|
|
`JOBS`) on the first attempt and one on the serial diagnostic retry. Rust's
|
|
bootstrap uses this job count for LLVM's compile pool and CMake build command.
|
|
|
|
Built the corrected `configured` image for mediatek/filogic 25.12.5, and
|
|
expanded the actual OpenWrt `Host/Compile` recipe at both job counts: it invokes
|
|
`x.py --jobs=8` and `x.py --jobs=1` respectively. The same recipe checks pass
|
|
against the 24.10.8 feed in a disposable container. ShellCheck, shell syntax,
|
|
actionlint and diff checks pass. Full source compilation after this change has
|
|
not been rerun, and no images were published or infrastructure limits changed.
|
|
|
|
## Follow-up: runner-owned Redis and SSD cache — 2026-10-11
|
|
|
|
Initially prototyped a 1 GiB Redis LRU hot cache and Apache WebDAV backed by a
|
|
persistent Docker volume on `fastpool`. This prototype was superseded by the
|
|
S3 setup below after testing both requested alternatives. The SDK repository
|
|
consumes these services and no longer deploys Redis. Production services have
|
|
not been redeployed; the original Redis container and volume remain untouched.
|
|
|
|
Inspected the pinned sccache 0.18.0 source and ran a cross-container central
|
|
server experiment. With client-side compilation and an unshared disk backend,
|
|
the first compile missed; the second reported a cache error and recompiled,
|
|
because `IpcStorage` attempts to open the daemon's returned absolute cache path.
|
|
TCP addresses are fixed to loopback in `net::SocketAddr::with_port`. In addition,
|
|
client-side `get_raw` bypasses multi-level read statistics and backfilling. The
|
|
standard per-job daemon mode with Redis/WebDAV avoids these restrictions.
|
|
|
|
Writes in standard mode can still be outstanding when the compiler client
|
|
returns. `SCCACHE_MULTILEVEL_WRITE_ERROR_POLICY=all` awaits both tiers within a
|
|
write; `sccache-stats` then drains writes and backfills for up to 30 seconds before
|
|
printing statistics and stopping the job daemon. Its completion checks use the
|
|
pinned version's cache-miss/write and per-tier backfill counters.
|
|
|
|
Passed on TheBeast using an isolated `sccache-probe` Compose project:
|
|
|
|
- Built the WebDAV image and validated Apache configuration. Real writes exposed
|
|
the base Alpine image's missing APR DBM driver; the added Dockerfile installs
|
|
`apr-util-dbm_gdbm`. Both services then passed health checks and real writes.
|
|
- Retained CI verification: one miss writing both tiers, one Redis hit in a
|
|
fresh container, then one WebDAV-only hit in a third fresh container. Zero
|
|
cache errors, read/write errors, or per-tier write failures.
|
|
- Real Rust compilation inside Docker BuildKit followed by a fresh SDK runtime
|
|
container: miss, then hit, using host-loopback and Docker-DNS endpoints.
|
|
- Repeated the unchanged BuildKit build: the actual compilation RUN step was
|
|
cached and did not invoke sccache again.
|
|
- Recreated both test service containers, retaining only the WebDAV volume:
|
|
Redis started empty. A fresh SDK container hit WebDAV and backfilled Redis;
|
|
the following fresh container hit Redis.
|
|
- Identical source and prefix with GNU versus musl targets: musl missed, then
|
|
hit. Rust 1.90.0 versus 1.94.0 under the same test prefix: 1.94.0 missed, then
|
|
hit. All cache-error counters remained zero.
|
|
- A disposable OpenWrt package compiled via the real `rust-package.mk`
|
|
`Build/Compile/Cargo` helper and SDK musl linker in two fresh SDK containers:
|
|
dependency miss, then hit. This used the actual completed 24.10.8 SDK image,
|
|
not the earlier validation toolchain. The binary remained non-cacheable.
|
|
- Proposed full runner Compose configuration, YAML parsing, Apache config,
|
|
ShellCheck, shell syntax, Python syntax, actionlint, and diff checks passed.
|
|
actionlint excludes only Gitea's built-in checkout syntax. Dockerfile checks
|
|
exclude the public `KEY_PREFIX` secret-name heuristic and mandatory base
|
|
image argument warning, as in the original validation.
|
|
- Publishing remains manual, with the original image tags and verification
|
|
before registry login/push. No login or publication was performed. Matrix,
|
|
runner labels/capacity, BuildKit caching, and serial verbose retry are unchanged.
|
|
|
|
Full SDK source bootstrap and a Gitea dispatch were not rerun for this cache
|
|
change. Deploy/build the updated runner stack before using the changed workflow.
|
|
The prototype WebDAV tier had no automatic LRU or capacity limit; the final
|
|
S3 configuration below adds an independent hard filesystem bound and retention.
|
|
Old Redis entries are not migrated; retiring the old service/volume is a separate
|
|
approved cutover step. Test logs are under `/tmp/gitea-sccache-stack` and
|
|
`/tmp/gitea-sccache-tests` on the runner; isolated test services are cleaned up.
|
|
|
|
Sources: [sccache architecture](https://github.com/mozilla/sccache/blob/v0.18.0/docs/Architecture.md),
|
|
[pinned IPC storage](https://github.com/mozilla/sccache/blob/v0.18.0/src/cache/ipc_storage.rs),
|
|
[pinned multi-level implementation](https://github.com/mozilla/sccache/blob/v0.18.0/src/cache/multilevel.rs).
|
|
|
|
|
|
## Follow-up: RustFS and VersityGW comparison — 2026-10-11
|
|
|
|
Selected **RustFS 1.0.1** after testing it and **VersityGW 1.8.0**, following the
|
|
owner's preference. The final runner Compose configuration uses Redis/S3 and
|
|
removes the uncommitted Apache image/config prototype. Services were tested in
|
|
isolated projects, not deployed over the existing runner/cache services.
|
|
|
|
Both candidates passed real sccache 0.18.0 compilation tests using the completed
|
|
SDK images on TheBeast:
|
|
|
|
- Cold miss wrote both tiers; fresh-container Redis hit; S3-only hit.
|
|
- Service restart plus empty isolated Redis: S3 hit backfilled Redis, then the
|
|
next fresh container hit Redis. Cache read/write/error counters stayed zero.
|
|
- Same prefix/input with GNU versus musl targets: musl missed then hit. Rust
|
|
1.90.0 versus 1.94.0: the different toolchain missed then hit.
|
|
|
|
The objects live on a separate Incus ZFS volume with a 16 GiB hard quota for
|
|
these tests. A separate 64 MiB quota probe rejected an 80 MiB incompressible write
|
|
with `Disk quota exceeded`. No production data was used for the quota test.
|
|
Production setup requires a new **100 GiB** quota-backed Incus volume; Compose
|
|
requires an existing data directory and rejects a filesystem reporting more than
|
|
100 GiB capacity. This guard passed on the quota-backed mount and rejected the
|
|
unbounded SSD filesystem. Missing BuildKit credentials failed before compilation,
|
|
and test logs contained no credential values. RustFS log files use a bounded tmpfs; Docker logs rotate.
|
|
|
|
Storage microbenchmark: 64 objects of 1 MiB incompressible data, three passes,
|
|
median aggregate MiB/s over host-loopback HTTP on the same SSD pool. Every GET
|
|
was compared with its source bytes. These include client/protocol overhead and
|
|
warm filesystem cache; they do not measure full SDK build speed or sustained
|
|
cold-SSD throughput.
|
|
|
|
| Backend | PUT, 1 client | GET, 1 client | PUT, 4 clients | GET, 4 clients |
|
|
| --- | ---: | ---: | ---: | ---: |
|
|
| Apache WebDAV prototype | 110.2 | 660.8 | failed | not run |
|
|
| RustFS 1.0.1 | 101.5 | 314.7 | 231.2 | 471.2 |
|
|
| VersityGW 1.8.0 | 99.1 | 329.3 | 232.2 | 410.2 |
|
|
|
|
WebDAV's four-client PUT test returned HTTP 500 lock-database errors, despite
|
|
passing sequential compiler-cache tests. Both S3 backends handled that load.
|
|
Observed Docker memory usage after the probes was about 202 MiB for RustFS and
|
|
28 MiB for VersityGW; these are small test workloads, not worst-case bounds.
|
|
RustFS was chosen for preference and compatibility, not a proven performance
|
|
advantage over VersityGW. Sequential S3 GETs were slower than the WebDAV baseline.
|
|
Redis remains the primary hot tier.
|
|
|
|
Final RustFS setup and SDK integration checks:
|
|
|
|
- The entire proposed runner Compose configuration validates with dummy
|
|
registration credentials. Only the cache services ran in the isolated project;
|
|
both passed their real health checks. Bucket initialization with pinned official
|
|
AWS CLI 2.37.12 passed. RustFS accepted and returned the exact configured
|
|
30-day object expiration and one-day abandoned multipart-upload policy.
|
|
Expiration execution itself was not tested by waiting 30 days.
|
|
- Replaced the original test RustFS container with the proposed Compose service
|
|
over the same retained quota-backed data: an earlier Rust artifact was still an
|
|
S3 hit and backfilled a new Redis namespace in a fresh SDK container.
|
|
- The retained `ci/verify-cache.sh` passed all five probes (cold writes, Redis hit,
|
|
S3-only hit, backfill into a fresh Redis namespace, subsequent Redis hit) on
|
|
x86-64/24.10.8 and mediatek-filogic/25.12.5. It never flushes shared Redis data.
|
|
- Real BuildKit Rust compilation using the proposed required secret mounts,
|
|
then a fresh runtime-container cache hit, passed. Secret values were absent
|
|
from exported image configuration/history; subsequent layers and a fresh
|
|
container had no credential environment variables or secret files. The repeated
|
|
unchanged compilation layer was cached. Only a local disposable image was
|
|
produced; no registry login/push occurred.
|
|
- Actual OpenWrt `Build/Compile/Cargo` and SDK musl linker compiled a disposable
|
|
package twice: dependency miss followed by hit in a fresh container. This
|
|
confirms inherited S3 settings/credentials reach OpenWrt's sccache wrapper.
|
|
- BuildKit checked the exact changed SDK Dockerfile. Only the known public
|
|
`KEY_PREFIX` secret-name heuristic and required base-image argument warning were
|
|
excluded. Workflow/actionlint, ShellCheck, shell/Python/JSON/YAML parsing and
|
|
whitespace checks passed.
|
|
|
|
The SDK image retains public S3 settings and version-only prefixes. Credentials are not stored in image configuration/history, and runner endpoints
|
|
are build arguments rather than runtime defaults. Gitea secrets feed ephemeral BuildKit
|
|
mounts and runtime verification containers. The runner service's registration,
|
|
labels/capacity, build matrix, image tags, manual publication gate, BuildKit layer
|
|
caching and serial verbose fallback are unchanged.
|
|
|
|
Manual setup: allocate/mount the 100 GiB volume, set the two cache credentials in
|
|
Komodo and Gitea, deploy the cache services, initialize the bucket, and run a
|
|
non-publishing workflow before cutover. Full SDK bootstrap and live Gitea dispatch
|
|
were not repeated for this backend change. Production Redis, its volume, and
|
|
Jenkins remain untouched pending acceptance/approved retirement. Isolated test
|
|
containers and volumes are removed after verification; logs remain under
|
|
`/tmp/s3-cache-tests` on the runner.
|
|
|
|
Sources: [RustFS deployment](https://docs.rustfs.com/en/installation/container/docker),
|
|
[VersityGW Docker setup](https://github.com/versity/versitygw/wiki/Docker),
|
|
[sccache S3](https://github.com/mozilla/sccache/blob/v0.18.0/docs/S3.md),
|
|
[BuildKit secrets](https://docs.docker.com/build/building/secrets/).
|