ci(cache): Migrate Rust cache to runner-managed multi-level Redis and S3

Centralize cache service management in the Gitea runner for improved persistence and reliability using an SSD-backed S3 backend. This ensures robust verification of multi-tier cache operations.

ci/VALIDATION.md

@@ -109,3 +109,161 @@ expanded the actual OpenWrt `Host/Compile` recipe at both job counts: it invokes
 against the 24.10.8 feed in a disposable container. ShellCheck, shell syntax,
 actionlint and diff checks pass. Full source compilation after this change has
 not been rerun, and no images were published or infrastructure limits changed.
+
+## Follow-up: runner-owned Redis and SSD cache — 2026-10-11
+
+Initially prototyped a 1 GiB Redis LRU hot cache and Apache WebDAV backed by a
+persistent Docker volume on `fastpool`. This prototype was superseded by the
+S3 setup below after testing both requested alternatives. The SDK repository
+consumes these services and no longer deploys Redis. Production services have
+not been redeployed; the original Redis container and volume remain untouched.
+
+Inspected the pinned sccache 0.18.0 source and ran a cross-container central
+server experiment. With client-side compilation and an unshared disk backend,
+the first compile missed; the second reported a cache error and recompiled,
+because `IpcStorage` attempts to open the daemon's returned absolute cache path.
+TCP addresses are fixed to loopback in `net::SocketAddr::with_port`. In addition,
+client-side `get_raw` bypasses multi-level read statistics and backfilling. The
+standard per-job daemon mode with Redis/WebDAV avoids these restrictions.
+
+Writes in standard mode can still be outstanding when the compiler client
+returns. `SCCACHE_MULTILEVEL_WRITE_ERROR_POLICY=all` awaits both tiers within a
+write; `sccache-stats` then drains writes and backfills for up to 30 seconds before
+printing statistics and stopping the job daemon. Its completion checks use the
+pinned version's cache-miss/write and per-tier backfill counters.
+
+Passed on TheBeast using an isolated `sccache-probe` Compose project:
+
+- Built the WebDAV image and validated Apache configuration. Real writes exposed
+  the base Alpine image's missing APR DBM driver; the added Dockerfile installs
+  `apr-util-dbm_gdbm`. Both services then passed health checks and real writes.
+- Retained CI verification: one miss writing both tiers, one Redis hit in a
+  fresh container, then one WebDAV-only hit in a third fresh container. Zero
+  cache errors, read/write errors, or per-tier write failures.
+- Real Rust compilation inside Docker BuildKit followed by a fresh SDK runtime
+  container: miss, then hit, using host-loopback and Docker-DNS endpoints.
+- Repeated the unchanged BuildKit build: the actual compilation RUN step was
+  cached and did not invoke sccache again.
+- Recreated both test service containers, retaining only the WebDAV volume:
+  Redis started empty. A fresh SDK container hit WebDAV and backfilled Redis;
+  the following fresh container hit Redis.
+- Identical source and prefix with GNU versus musl targets: musl missed, then
+  hit. Rust 1.90.0 versus 1.94.0 under the same test prefix: 1.94.0 missed, then
+  hit. All cache-error counters remained zero.
+- A disposable OpenWrt package compiled via the real `rust-package.mk`
+  `Build/Compile/Cargo` helper and SDK musl linker in two fresh SDK containers:
+  dependency miss, then hit. This used the actual completed 24.10.8 SDK image,
+  not the earlier validation toolchain. The binary remained non-cacheable.
+- Proposed full runner Compose configuration, YAML parsing, Apache config,
+  ShellCheck, shell syntax, Python syntax, actionlint, and diff checks passed.
+  actionlint excludes only Gitea's built-in checkout syntax. Dockerfile checks
+  exclude the public `KEY_PREFIX` secret-name heuristic and mandatory base
+  image argument warning, as in the original validation.
+- Publishing remains manual, with the original image tags and verification
+  before registry login/push. No login or publication was performed. Matrix,
+  runner labels/capacity, BuildKit caching, and serial verbose retry are unchanged.
+
+Full SDK source bootstrap and a Gitea dispatch were not rerun for this cache
+change. Deploy/build the updated runner stack before using the changed workflow.
+The prototype WebDAV tier had no automatic LRU or capacity limit; the final
+S3 configuration below adds an independent hard filesystem bound and retention.
+Old Redis entries are not migrated; retiring the old service/volume is a separate
+approved cutover step. Test logs are under `/tmp/gitea-sccache-stack` and
+`/tmp/gitea-sccache-tests` on the runner; isolated test services are cleaned up.
+
+Sources: [sccache architecture](https://github.com/mozilla/sccache/blob/v0.18.0/docs/Architecture.md),
+[pinned IPC storage](https://github.com/mozilla/sccache/blob/v0.18.0/src/cache/ipc_storage.rs),
+[pinned multi-level implementation](https://github.com/mozilla/sccache/blob/v0.18.0/src/cache/multilevel.rs).
+
+
+## Follow-up: RustFS and VersityGW comparison — 2026-10-11
+
+Selected **RustFS 1.0.1** after testing it and **VersityGW 1.8.0**, following the
+owner's preference. The final runner Compose configuration uses Redis/S3 and
+removes the uncommitted Apache image/config prototype. Services were tested in
+isolated projects, not deployed over the existing runner/cache services.
+
+Both candidates passed real sccache 0.18.0 compilation tests using the completed
+SDK images on TheBeast:
+
+- Cold miss wrote both tiers; fresh-container Redis hit; S3-only hit.
+- Service restart plus empty isolated Redis: S3 hit backfilled Redis, then the
+  next fresh container hit Redis. Cache read/write/error counters stayed zero.
+- Same prefix/input with GNU versus musl targets: musl missed then hit. Rust
+  1.90.0 versus 1.94.0: the different toolchain missed then hit.
+
+The objects live on a separate Incus ZFS volume with a 16 GiB hard quota for
+these tests. A separate 64 MiB quota probe rejected an 80 MiB incompressible write
+with `Disk quota exceeded`. No production data was used for the quota test.
+Production setup requires a new **100 GiB** quota-backed Incus volume; Compose
+requires an existing data directory and rejects a filesystem reporting more than
+100 GiB capacity. This guard passed on the quota-backed mount and rejected the
+unbounded SSD filesystem. Missing BuildKit credentials failed before compilation,
+and test logs contained no credential values. RustFS log files use a bounded tmpfs; Docker logs rotate.
+
+Storage microbenchmark: 64 objects of 1 MiB incompressible data, three passes,
+median aggregate MiB/s over host-loopback HTTP on the same SSD pool. Every GET
+was compared with its source bytes. These include client/protocol overhead and
+warm filesystem cache; they do not measure full SDK build speed or sustained
+cold-SSD throughput.
+
+| Backend | PUT, 1 client | GET, 1 client | PUT, 4 clients | GET, 4 clients |
+| --- | ---: | ---: | ---: | ---: |
+| Apache WebDAV prototype | 110.2 | 660.8 | failed | not run |
+| RustFS 1.0.1 | 101.5 | 314.7 | 231.2 | 471.2 |
+| VersityGW 1.8.0 | 99.1 | 329.3 | 232.2 | 410.2 |
+
+WebDAV's four-client PUT test returned HTTP 500 lock-database errors, despite
+passing sequential compiler-cache tests. Both S3 backends handled that load.
+Observed Docker memory usage after the probes was about 202 MiB for RustFS and
+28 MiB for VersityGW; these are small test workloads, not worst-case bounds.
+RustFS was chosen for preference and compatibility, not a proven performance
+advantage over VersityGW. Sequential S3 GETs were slower than the WebDAV baseline.
+Redis remains the primary hot tier.
+
+Final RustFS setup and SDK integration checks:
+
+- The entire proposed runner Compose configuration validates with dummy
+  registration credentials. Only the cache services ran in the isolated project;
+  both passed their real health checks. Bucket initialization with pinned official
+  AWS CLI 2.37.12 passed. RustFS accepted and returned the exact configured
+  30-day object expiration and one-day abandoned multipart-upload policy.
+  Expiration execution itself was not tested by waiting 30 days.
+- Replaced the original test RustFS container with the proposed Compose service
+  over the same retained quota-backed data: an earlier Rust artifact was still an
+  S3 hit and backfilled a new Redis namespace in a fresh SDK container.
+- The retained `ci/verify-cache.sh` passed all five probes (cold writes, Redis hit,
+  S3-only hit, backfill into a fresh Redis namespace, subsequent Redis hit) on
+  x86-64/24.10.8 and mediatek-filogic/25.12.5. It never flushes shared Redis data.
+- Real BuildKit Rust compilation using the proposed required secret mounts,
+  then a fresh runtime-container cache hit, passed. Secret values were absent
+  from exported image configuration/history; subsequent layers and a fresh
+  container had no credential environment variables or secret files. The repeated
+  unchanged compilation layer was cached. Only a local disposable image was
+  produced; no registry login/push occurred.
+- Actual OpenWrt `Build/Compile/Cargo` and SDK musl linker compiled a disposable
+  package twice: dependency miss followed by hit in a fresh container. This
+  confirms inherited S3 settings/credentials reach OpenWrt's sccache wrapper.
+- BuildKit checked the exact changed SDK Dockerfile. Only the known public
+  `KEY_PREFIX` secret-name heuristic and required base-image argument warning were
+  excluded. Workflow/actionlint, ShellCheck, shell/Python/JSON/YAML parsing and
+  whitespace checks passed.
+
+The SDK image retains public S3 settings and version-only prefixes. Credentials are not stored in image configuration/history, and runner endpoints
+are build arguments rather than runtime defaults. Gitea secrets feed ephemeral BuildKit
+mounts and runtime verification containers. The runner service's registration,
+labels/capacity, build matrix, image tags, manual publication gate, BuildKit layer
+caching and serial verbose fallback are unchanged.
+
+Manual setup: allocate/mount the 100 GiB volume, set the two cache credentials in
+Komodo and Gitea, deploy the cache services, initialize the bucket, and run a
+non-publishing workflow before cutover. Full SDK bootstrap and live Gitea dispatch
+were not repeated for this backend change. Production Redis, its volume, and
+Jenkins remain untouched pending acceptance/approved retirement. Isolated test
+containers and volumes are removed after verification; logs remain under
+`/tmp/s3-cache-tests` on the runner.
+
+Sources: [RustFS deployment](https://docs.rustfs.com/en/installation/container/docker),
+[VersityGW Docker setup](https://github.com/versity/versitygw/wiki/Docker),
+[sccache S3](https://github.com/mozilla/sccache/blob/v0.18.0/docs/S3.md),
+[BuildKit secrets](https://docs.docker.com/build/building/secrets/).

ci/build-rust.sh

@@ -2,11 +2,14 @@
 set -eu
 
 : "${SCCACHE_REDIS_ENDPOINT:?Set the persistent sccache Redis endpoint}"
+: "${SCCACHE_ENDPOINT:?Set the persistent sccache S3 endpoint}"
+: "${AWS_ACCESS_KEY_ID:?Supply the S3 access key through a BuildKit secret}"
+: "${AWS_SECRET_ACCESS_KEY:?Supply the S3 secret key through a BuildKit secret}"
 : "${JOBS:=8}"
 case "$JOBS" in ''|*[!0-9]*|0) echo 'JOBS must be a positive integer' >&2; exit 1 ;; esac
 
 sccache --start-server
-trap 'sccache --show-stats; sccache --stop-server' EXIT
+trap 'sccache-stats; sccache --stop-server' EXIT
 sccache --zero-stats
 # Bootstrap/Ninja does not inherit Make's job count. Bound both explicitly.
 make package/rust/host/compile -j"$JOBS" RUST_BUILD_JOBS="$JOBS" || \

ci/cache.compose.yml

@@ -1,40 +0,0 @@
-services:
-  cache:
-    image: redis:7.4.6-alpine
-    container_name: openwrt-rust-cache
-    restart: unless-stopped
-    command:
-      - redis-server
-      - --appendonly
-      - 'yes'
-      - --appendfsync
-      - everysec
-      - --maxmemory
-      - 6gb
-      - --maxmemory-policy
-      - allkeys-lru
-      - --save
-      - ''
-    mem_limit: 8g
-    memswap_limit: 8g
-    cpus: 1
-    ports:
-      - '127.0.0.1:6379:6379'
-    volumes:
-      - cache:/data
-    networks:
-      - jobs
-    healthcheck:
-      test: ['CMD', 'redis-cli', 'ping']
-      interval: 5s
-      timeout: 3s
-      retries: 12
-
-volumes:
-  cache:
-    name: openwrt-rust-sccache
-
-networks:
-  jobs:
-    external: true
-    name: gitea-runner-jobs

ci/sccache-stats.py

@@ -0,0 +1,26 @@
+#!/usr/bin/env python3
+"""Wait for this job's cache writes/backfills, then print complete statistics."""
+import json
+import subprocess
+import time
+
+end = time.monotonic() + 30
+while True:
+    data = json.loads(subprocess.check_output(
+        ["sccache", "--show-stats", "--stats-format", "json"],
+        timeout=max(0.1, end - time.monotonic()),
+    ))
+    stats = data["stats"]
+    misses = sum(stats["cache_misses"]["counts"].values())
+    levels = stats.get("multi_level") or []
+    writes_done = stats["cache_writes"] + stats["cache_write_errors"] >= misses
+    backfills_done = sum(x["backfills_to"] for x in levels) >= sum(
+        x["backfills_from"] for x in levels
+    )
+    if writes_done and backfills_done:
+        break
+    if time.monotonic() >= end:
+        subprocess.run(["sccache", "--show-stats"], check=True, timeout=5)
+        raise SystemExit("Timed out waiting for sccache writes/backfills")
+    time.sleep(0.1)
+subprocess.run(["sccache", "--show-stats"], check=True, timeout=5)

ci/setup-builder.sh

@@ -1,6 +1,5 @@
 #!/bin/sh
 set -eu
-docker compose -p openwrt-rust -f ci/cache.compose.yml up -d --wait
 if ! docker buildx inspect openwrt-sdk-rust >/dev/null 2>&1; then
     docker buildx create --name openwrt-sdk-rust --driver docker-container \
         --driver-opt network=host,memory=20g,memory-swap=20g,cpu-period=100000,cpu-quota=800000 \

ci/verify-cache.sh

@@ -1,16 +1,34 @@
 #!/bin/sh
-# Compile in two disposable containers: the second must read the first's cache.
+# Verify writes, Redis/S3 hits, and Redis backfilling in fresh containers.
 set -eu
 image=${1:?Usage: verify-cache.sh IMAGE}
+: "${SCCACHE_REDIS_ENDPOINT:=redis://sccache-redis:6379}"
+: "${SCCACHE_ENDPOINT:=http://sccache-s3:9000}"
+: "${AWS_ACCESS_KEY_ID:?Set the S3 access key}"
+: "${AWS_SECRET_ACCESS_KEY:?Set the S3 secret key}"
+stats_script=$(cat ci/sccache-stats.py)
 nonce=$(mktemp -d)
 trap 'rmdir "$nonce"' EXIT
-prefix="validation:${image##*:}:$(basename "$nonce"):"
-for expected in miss hit; do
+prefix="validation/${image##*:}/$(basename "$nonce")/"
+for expected in miss hit s3-hit backfill backfill-hit; do
+    redis_prefix=$prefix
+    case "$expected" in backfill*) redis_prefix="${prefix}backfill/" ;; esac
+    chain=redis,s3
+    if [ "$expected" = s3-hit ]; then chain=s3; fi
     docker run --rm -i --network gitea-runner-jobs --cpus 2 \
         --memory 2g --memory-swap 2g \
-        -e SCCACHE_REDIS_ENDPOINT=redis://openwrt-rust-cache:6379 \
-        -e SCCACHE_REDIS_KEY_PREFIX="$prefix" -e EXPECTED="$expected" \
-        "$image" sh -es <<'PROBE'
+        -e SCCACHE_REDIS_ENDPOINT="$SCCACHE_REDIS_ENDPOINT" \
+        -e SCCACHE_ENDPOINT="$SCCACHE_ENDPOINT" \
+        -e SCCACHE_REDIS_KEY_PREFIX="$redis_prefix" \
+        -e SCCACHE_S3_KEY_PREFIX="$prefix" \
+        -e SCCACHE_BUCKET=sccache -e SCCACHE_REGION=us-east-1 \
+        -e SCCACHE_S3_USE_SSL=false \
+        -e AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY \
+        -e SCCACHE_MULTILEVEL_CHAIN="$chain" \
+        -e SCCACHE_MULTILEVEL_WRITE_ERROR_POLICY=all \
+        -e SCCACHE_CLIENT_SIDE=0 \
+        -e SCCACHE_STATS_SCRIPT="$stats_script" \
+        -e EXPECTED="$expected" "$image" sh -es <<'PROBE'
 rustc=$(find /builder/staging_dir -type f -path '*/host/bin/rustc')
 test -x "$rustc"
 mkdir /tmp/sccache-probe
@@ -21,19 +39,29 @@ trap 'sccache --stop-server' EXIT
 sccache --zero-stats
 sccache "$rustc" --crate-name ci_sccache_probe \
     --crate-type rlib --emit link,dep-info --out-dir . probe.rs
-sccache --show-stats
+python3 -c "$SCCACHE_STATS_SCRIPT"
 sccache --show-stats --stats-format json > stats.json
 python3 - <<'PY'
 import json
 import os
 
 data = json.load(open("stats.json"))
-assert data["cache_location"].startswith("redis,"), data
+assert data["cache_location"].startswith("Multi-level"), data
 stats = data["stats"]
-field = "cache_hits" if os.environ["EXPECTED"] == "hit" else "cache_misses"
+expected = os.environ["EXPECTED"]
+field = "cache_misses" if expected == "miss" else "cache_hits"
 assert sum(stats[field]["counts"].values()) == 1, data
 assert stats["cache_read_errors"] == stats["cache_write_errors"] == 0, data
-print("Verified persistent cache", os.environ["EXPECTED"])
+assert sum(stats["cache_errors"]["counts"].values()) == 0, data
+levels = stats["multi_level"]
+assert all(level["write_failures"] == 0 for level in levels), data
+if expected == "miss":
+    assert len(levels) == 2 and all(level["writes"] == 1 for level in levels), data
+elif expected == "backfill":
+    assert levels[1]["hits"] == levels[0]["backfills_to"] == 1, data
+else:
+    assert levels[0]["hits"] == 1, data
+print("Verified persistent cache", expected)
 PY
 PROBE
 done