openwrt-sdk-rust

Builds the OpenWrt SDK with its Rust host toolchain installed. Gitea Actions resolves stable_version and oldstable_version from OpenWrt's .versions.json. The default targets remain x86/64,mediatek/filogic,octeon/generic; dispatch with targets=all to use upstream dump-target-info.pl, or supply a comma-separated list. Images still use ghcr.io/openwrt/sdk:<target-with-dashes>-<version> and publish to lochnair/openwrt-sdk-rust:<target-with-dashes>-<version> on Docker Hub. No latest tag or additional registry is introduced.

Runner setup and cutover

The workflow uses ubuntu-latest on the existing TheBeast runner, currently the only runner. Its job image and access to the persistent Docker daemon were verified; no additional runner, label, capacity change, or restart is required. builtin:checkout uses the existing Runner 5 checkout support.

Enable repository Actions and set DOCKERHUB_USERNAME and DOCKERHUB_TOKEN secrets with push permission to lochnair/openwrt-sdk-rust. Jenkins' credential ID is not reusable in Gitea. The workflow is manually dispatched, matching the parameterized Jenkins job; there are no PR, push, or scheduled publication triggers. publish=false is the safe validation default. Dispatch publish=true only after publication is approved. Login uses a temporary job directory, and each successfully built and checked image is pushed under its original tag. Before any push, ci/verify-cache.sh uses the installed SDK rustc in fresh containers and requires writes to both tiers, Redis/S3 hits, and backfilling.

Also set SCCACHE_S3_ACCESS_KEY_ID and SCCACHE_S3_SECRET_ACCESS_KEY from the runner cache service in Gitea repository or organization secrets. These are needed for non-publishing builds too; Docker Hub credentials are only used for publishing.

Run a non-publishing dispatch for the default six combinations, then an approved publishing dispatch. Check the pulled image's Rust/Cargo installation and downstream package build before disabling the old Jenkins job and deleting Jenkinsfile. Do not retire Jenkins before this acceptance run. No Jenkins plugins or credentials are used by the new workflow.

Persistent compilation cache

The Gitea runner's homelab-stacks/stacks/gitea-runner/compose.yaml owns Redis and the SSD-backed RustFS S3 service. This repository only prepares its reusable BuildKit builder and consumes those services; jobs never deploy them. Redis is a 1 GiB LRU hot cache with a 1.5 GiB memory limit. It is deliberately non-persistent: S3 holds the persistent copies on a quota-limited 100 GiB Incus/ZFS volume. Jobs, Redis restarts, and RustFS container recreation do not clear it. The cache bucket expires objects after 30 days; the filesystem quota provides an independent hard limit. See the runner stack's README for allocation and setup.

Each job runs its own pinned sccache 0.18.0 daemon with SCCACHE_MULTILEVEL_CHAIN=redis,s3 in the standard server-side mode. Writes go to both tiers; Redis misses fall through to S3 and backfill Redis. SCCACHE_MULTILEVEL_WRITE_ERROR_POLICY=all makes each cache write await both tiers. The installed sccache-stats helper waits up to 30 seconds for outstanding writes and backfills before printing statistics and stopping the job's daemon. BuildKit's layer cache remains separate.

Both services join gitea-runner-jobs. SDK/package containers use redis://sccache-redis:6379 and http://sccache-s3:9000; BuildKit's host network uses the loopback ports 6380 and 9000. There is no public ingress. S3 credentials are supplied through required BuildKit secret mounts for compilation and runtime environment variables for disposable verification/package containers. Credentials are not stored in the published SDK image. Runner endpoints are build arguments rather than default runtime settings. The image contains only public backend configuration: bucket sccache, region us-east-1, path-style HTTP, and cache prefixes. Other environments can override these values.

Each OpenWrt release has a shared prefix across targets: openwrt-rust:<version>: in Redis and openwrt-rust/<version>/ in S3. Compiler identities, target, flags, sources, and dependencies still determine cache keys, preventing incompatible reuse. The old Redis service and its volume remain untouched during this change; their entries are not copied into the new SSD cache.

A single persistent sccache daemon was tested across containers with client-side compilation. The second container reported a cache error and recompiled: the daemon's disk backend returns an absolute cache-file path that the client must open, but the disposable container cannot see that filesystem. Version 0.18.0 also fixes TCP connections to loopback, and client-side raw reads bypass multi-level backfilling and per-tier read statistics. Per-job daemons with remote storage avoid filesystem sharing, socket proxies, and job-network changes.

RustFS and VersityGW were both tested with real Rust compilation, S3 fallback, Redis backfilling, container restarts, and differing targets/toolchains. Both passed. RustFS was selected following the owner's preference; the small storage benchmark does not establish a full-build speedup. Deploy the runner cache and initialize its bucket before running this workflow.

The shared BuildKit worker executes one build step at a time, even across simultaneous dispatches. It is limited to eight CPUs and 20 GiB RAM with no swap. Make and Rust bootstrap both receive eight jobs; the original -j1 V=sc diagnostic retry also reduces bootstrap/Ninja to one job. Rust build-directory/archive cleanup remains. BuildKit layer storage has a 50 GiB GC budget. Matrix jobs also run one at a time and stop scheduling on a failure, as Jenkins' sequential stage loop did.

OpenWrt integration was inspected in the SDK's pinned packages feed:

• rust-values.mk unexports RUSTC_WRAPPER. Setting it in the workflow alone does not enable caching. CONFIG_RUST_SCCACHE=y makes OpenWrt inject CARGO_INCREMENTAL=0, RUSTC_WRAPPER=sccache, and SCCACHE_DIR into host and package Cargo invocations. The Dockerfile enables and checks that option.
• The base SDK has no sccache binary. We install the pinned 0.18.0 amd64-musl release and verify its SHA-256 before using it.
• Rust bootstrap wraps rustc itself. We also set build.ccache=sccache in its configure arguments so stage-zero compilation and LLVM use sccache, retaining llvm.download-ci-llvm=false. The inherited wrapper is forwarded by bootstrap through RUSTC_WRAPPER_REAL where applicable.
• sccache is a compiler cache, not a Cargo target-directory cache. Linked binaries, proc macros, some bootstrap work and incremental compilation are not cacheable. SDK bootstrap and downstream dependencies can share storage without necessarily sharing every compiled object. Every actual SDK build prints statistics on success or failure. BuildKit can reuse unchanged layers; a cached compilation layer does not run sccache or produce fresh statistics.

Sources: OpenWrt Rust configuration, Rust bootstrap, sccache multi-level cache, sccache client-side architecture, Rust cache limitations.

Downstream OpenWrt packages

Use the same cache services and target/version image. Export AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY from CI secrets before starting the container. For example, after installing the sqm-autorate-rust feed/package in the SDK:

docker run --rm --network gitea-runner-jobs --cpus 8 \
  --memory 20g --memory-swap 20g \
  -e SCCACHE_REDIS_ENDPOINT=redis://sccache-redis:6379 \
  -e SCCACHE_ENDPOINT=http://sccache-s3:9000 \
  -e AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY \
  lochnair/openwrt-sdk-rust:x86-64-24.10.8 sh -ec '
    # Install/update the downstream feed here before defconfig.
    make defconfig
    grep -qx "CONFIG_RUST_SCCACHE=y" .config
    sccache --start-server
    trap "sccache-stats; sccache --stop-server" EXIT
    sccache --zero-stats
    make package/sqm-autorate-rust/compile -j8 V=sc
  '

Keep the image's cache prefix and enable CONFIG_RUST_SCCACHE after any downstream configuration replacement. Cache statistics belong to the current container's sccache server; --zero-stats does not erase cached objects. sqm-autorate-rust currently uses cross-rs to publish static binaries, not an OpenWrt SDK package workflow; that unrelated pipeline is unchanged. The package Makefile in openwrt-feeds already uses OpenWrt's rust-package.mk, so an SDK package job can use this cache without patching the package.

For a local SDK build on the prepared runner, export AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY from the cache credentials in your environment first:

sh ci/setup-builder.sh
docker buildx build --builder openwrt-sdk-rust --load --progress plain \
  --network host --allow network.host \
  --build-arg SAFE_TARGET=x86-64 --build-arg VERSION=24.10.8 \
  --build-arg JOBS=8 \
  --build-arg SCCACHE_REDIS_ENDPOINT=redis://127.0.0.1:6380 \
  --build-arg SCCACHE_ENDPOINT=http://127.0.0.1:9000 \
  --secret id=aws-access-key-id,env=AWS_ACCESS_KEY_ID \
  --secret id=aws-secret-access-key,env=AWS_SECRET_ACCESS_KEY \
  -t lochnair/openwrt-sdk-rust:x86-64-24.10.8 .

See validation results for the checks performed and the remaining cutover acceptance.