Container bases and toolchain versions¶

Shared container bases and toolchain versions are defined in build-config.env at the repository root. The contract covers the entries declared there; compatibility test versions need separate review before consolidation.

To change one of these shared settings in a fork, edit that file and regenerate its mirrors.

Quick start¶

$EDITOR build-config.env      # change a pin
make base-images-sync         # push it into every Dockerfile
git diff                      # review, then commit both

make base-images-sync rewrites the ARG defaults in every Dockerfile from the config and then re-runs the check, so a clean run means the tree agrees with the config.

How it works¶

No Dockerfile names a base image directly. Each one takes it as a build argument whose default mirrors build-config.env:

ARG RELEASE_RUNTIME_CC="gcr.io/distroless/cc-debian13:nonroot@sha256:c31ff9ab…"

FROM ${RELEASE_RUNTIME_CC} AS runtime-base

Two consequences worth knowing:

• A plain docker build still works. The default is a real value, so you do not need a wrapper script or a bake file to build any image in this repo.
• CI can override any base with --build-arg RELEASE_RUNTIME_CC=… without editing a Dockerfile — useful for testing a candidate base before pinning it.

The single-source gate¶

The mirrored defaults are what would rot, so scripts/ci/check-base-image-single-source.sh fails the build if any of them drifts from the config. It runs in make lint-sh and as a pre-commit hook.

Vendor libraries come from named stages¶

Pulling a library straight out of a vendor image looks harmless:

COPY --from=nvidia/cuda:<tag>@sha256:… /usr/local/cuda/lib64/libcudart.so* /usr/local/lib/

but that is a base-image pin — it decides which CUDA runtime the shipped image carries — and it is invisible to anyone grepping for FROM. Four such pins in this repo were the most out-of-date things in it. Declare a named stage instead:

FROM ${CUDA_RUNTIME} AS cuda-runtime-libs

COPY --from=cuda-runtime-libs /usr/local/cuda/lib64/libcudart.so* /usr/local/lib/

BuildKit prunes the stage when the selected target does not use it. The gate rejects direct external FROM and COPY --from references with or without a digest: alpine, alpine:latest and alpine@sha256:… all need a centrally owned named stage. Instruction case, --platform, other COPY flags and continued instructions do not exempt a reference. Docker documents these forms in its Dockerfile reference.

Declare each shared image's global ARG NAME=value on one physical line before the first FROM, so the mirror checker and --write can maintain it. FROM ${NAME} or FROM $NAME must use that declared configuration key; an arbitrary new ARG or a fallback such as ${NAME:-alpine} is rejected. Use named stages or an earlier numeric stage index for COPY --from.

ROCm and oneAPI bases¶

The ROCm builder and runtime source use AMD's released Ubuntu 26.04 10.1.0-full image, pinned through ROCM_BUILDER and ROCM_RUNTIME. The published ROCm image and the AMD GPU tester image do not build FROM it: their Debian 13 builders stream /opt/rocm out of ROCM_BUILDER with scripts/ci/install-rocm-from-image.sh, and their runtimes copy only the HIP runtime files vmaf loads (ADR-1517).

• The rocm-src stage compiles and links a small HIP kernel after pruning the SDK, then runs its host-only entry point. This checks the compiler and loader without requiring an AMD GPU; device execution remains a separate test.
• A ROCm update changes more than the two pins. ROCm installs under /opt/rocm/core-<major>.<minor>, which the rocm-src stage and docker/Dockerfile.node name; tools/rc1-tester/image/hip-runtime.json names the LLVM library sonames (libLLVM.so.24.0git in 10.1.0); and tools/rc1-tester/image/licensing.json names the TheRock, rocm-systems and llvm-project commits of the release's share/therock/therock_manifest.json. The compiler can move with it (10.1.0 ships AMD clang 24, 10.0.0 shipped 23), so the HIP device suite runs again on a GPU before the update lands.
• The node runtime retains the vendor library directory structure when copying the HIP dependency closure into Debian 13. See the 26.04 verification.

oneAPI moved to Debian 13 with Intel's apt packages (ADR-1368); see Deliberate exceptions.

The two tracks¶

Keep both on the same libc generation unless you have a written reason not to. A dev container on a different libc than the release image tests the wrong thing.

The native Linux release bundle (libvmaf.so* and vmaf) follows the release track even though it is built from dev/Containerfile: its release-build stage takes RELEASE_BUILDER_BASE, not DEV_BASE (ADR-1354). A binary compiled on the Ubuntu 26.04 dev base needs glibc 2.43 and does not start on Debian 13, on Ubuntu 24.04 or on RELEASE_RUNTIME_CC.

Version knobs¶

The top of the config carries the human-meaningful version of each pin (RELEASE_DEBIAN, GO_VERSION, CUDA_VERSION, …). The gate asserts that each pinned tag actually carries the version its knob claims, so RELEASE_DEBIAN=13 cannot sit above a debian:12 pin. That check is what would have caught the drift this file exists to prevent.

CUDA: a coordinated pin, not an image tag¶

CUDA is the one knob whose value is coordinated across configuration and Dockerfiles. Following ADR-1300 and ADR-1306, the fork dropped Jimver/cuda-toolkit and all nvidia/cuda base images. CUDA builders and runtimes build FROM digest-pinned Ubuntu 26.04 (CUDA_BUILDER and CUDA_RUNTIME in build-config.env) and install the version-locked toolkit via scripts/ci/install-cuda-toolkit.sh (--mode=builder, --mode=runtime, or --mode=full). The two CUDA bases must equal DEV_BASE exactly, including its digest; the same owner also writes the narrow docker/dev/ubuntu-26.04-cuda.Dockerfile mirror. This decouples CUDA release bumps from upstream NVIDIA OCI image publication latency. The published CUDA image and the NVIDIA GPU tester image build on the release track's Debian 13 (RELEASE_BUILDER_BASE) instead, with the same script installing nvcc from NVIDIA's debian13 repository, and their runtimes hold no NVIDIA file (ADR-1517).

One release, seven spellings¶

One release is named in seven places across two files:

scripts/ci/check-cuda-pin-lockstep.py inventories all seven on every commit, checks the release-valued sites against CUDA_VERSION, requires the component versions to remain in its major/minor series, and fails on a CUDA release literal in any spelling it does not recognise (including any reintroduced nvidia/cuda image tag), so an untracked copy cannot appear quietly. The component build numbers are not derivable from the marketing release: NVIDIA shipped different nvcc and cudart builds within 13.4. The installer therefore uses exact package=version apt operands and verifies every installed version with dpkg-query.

Moving the release¶

The Windows CI legs read the same CUDA_VERSION: scripts/ci/install-cuda-toolkit.ps1 installs the toolkit from NVIDIA's redistributable manifest for that release.

To move the release:

1. Edit CUDA_VERSION in build-config.env, then derive CUDA_APT_PACKAGE and the runtime label:

   make cuda-pin-sync
   

2. Verify NVIDIA's redist manifest and the ubuntu2604 Packages index, then update CUDA_APT_LOCK_RELEASE and the three exact *_VERSION values by hand.

3. Synchronise the Dockerfile mirrors, which ensures they match build-config.env:

   make base-images-sync
   

Renovate discovers new CUDA releases from NVIDIA's official redist HTML index through custom.nvidia-cuda-redist, not from the retired nvidia/cuda image tags, and proposes bumping CUDA_VERSION directly. Only redistrib_X.Y.Z.json links are accepted. The index has no release timestamps, so the datasource-specific rule is timestamp-optional but stays manual-review and non-automerge. make cuda-pin-sync derives the mechanical spellings. Changing only CUDA_VERSION intentionally leaves CUDA_APT_LOCK_RELEASE stale, so the bot PR stays red until the exact metadata review is complete. See ADR-1285, ADR-1300, and ADR-1306.

Formatter versions¶

.pre-commit-config.yaml owns the ruff and black versions. The Makefile repeats them as RUFF_VERSION and BLACK_VERSION because make lint-tools installs the same tools into the project environment, and a formatter that differs between the hook and make lint disagrees about what counts as a violation. The gate compares the two files and rejects a literal ruff== or black== in a recipe, so a recipe can only name the variable. Renovate raises the Makefile pins in the same pull request as the hook revisions.

The type checker's Python version¶

pyproject.toml declares the project's Python floor once, as [project] requires-python, and build-config.env carries the interpreter CI installs as PYTHON_CI_VERSION. [tool.mypy] python_version has to agree with both: mypy checks the language version it is told to model, not the one it runs on. The gate compares all three and fails on a mismatch, on a python_version given as a bare number instead of a quoted string, and on the key being deleted — deleting it makes mypy follow whatever interpreter the caller happens to have, which is the same drift by another route.

It is gated rather than commented because the comment did not hold: the pin sat at 3.10 against a >=3.14 floor until ADR-1282. Below 3.12 mypy refuses to parse the PEP 695 type statement in numpy's bundled __init__.pyi, and that one blocking [syntax] error aborts the whole ai/src/ pass before any source file is checked, so the mis-modelled version silently disabled the check in every checkout that had numpy installed. Raise all three values in one commit.

Python and ONNX Runtime ownership¶

Scientific Python dependency floors remain in each package's pyproject.toml. For the classic vmaf package, python/pyproject.toml owns runtime dependencies; make python-deps-sync derives python/requirements.txt, and the requirements single-source gate rejects drift. The AI and MCP manifests own their own runtime and optional dependencies. Build-system requirements are separate metadata and must retain dependency updates already merged into the base.

The five scientific-stack globals originally proposed in ADR-1236 had no consumers or drift checks. They are deferred until that consumption is wired; adding a declaration alone does not move ownership out of package metadata.

Native ONNX Runtime pins also represent different contracts:

Preserve those lane roles when consolidating pins. An upgrade must verify the exact archive name, runtime-library closure and relevant tests: the 1.29 GPU release names select gpu_cuda12 or gpu_cuda13, while the 1.22 coverage URL uses gpu. A version-only replacement therefore does not preserve the download contract. None of these lane pins defines every Python package's ORT floor.

Everything is digest-pinned¶

A tag alone is not a pin: it moves under you, and reproducing a release build six months later is the whole point. The gate rejects any entry without @sha256:. Renovate updates the digests in place — see the docker manager in renovate.json.

Deliberate exceptions¶

• Local image consumers: Dockerfile.ffmpeg extends vmaf:latest, built from the root Dockerfile. dev/Containerfile.runner uses ARG BASE_IMAGE=vmaf-dev-mcp:local, built from dev/Containerfile. These exceptions bind the exact file, argument (where applicable) and value. An unpinned tag in any other consumer is not assumed to be local.
• docker/dev/*.Dockerfile pin Alpine, Arch and Fedora on purpose. They exist to prove the build survives distros the release track does not use, so unifying their bases would defeat them. The gate skips that directory.
• No distro exemptions remain. ROCm and oneAPI were once exempt from the "no Ubuntu 24.04" rule because each needed an SDK migration, not a pin swap (ADR-1231, research digest). ROCm moved to its Ubuntu 26.04 image; oneAPI moved to Debian 13 with Intel's apt packages, and the gate requires ONEAPI_BUILDER and ONEAPI_RUNTIME to equal RELEASE_BUILDER_BASE, as it requires the CUDA bases to equal DEV_BASE (ADR-1368).

Adding a new image¶

1. Add a semantic entry to build-config.env — name it for its role (RELEASE_RUNTIME_CC), not for the file that uses it.
2. Reference it as ARG + FROM ${…} in the Dockerfile.
3. Run make base-images-sync.

Verify the guard¶

python3 -m unittest discover -s scripts/ci/tests -p 'test_*single_source.py' -v
bash scripts/ci/check-base-image-single-source.sh

The tests run the actual gate in temporary Git repositories, without builds or registry access. They cover external image bypasses, local exceptions, named stages and mirror repair. The Level Zero fixtures also execute the container download command with temporary command stubs and an altered config, so no package installation or network access is needed. The pre-commit regression hook runs when the guard or configuration changes; CI's all-files pre-commit run includes it.

If two Dockerfiles want the same image, they share one entry. That collapsing is the point: every base image resolves to one build-config.env pin, and make base-images-sync lists them.

CI workflows read the same file¶

Version pins are not only in Dockerfiles. Before this file, ROCm was 7.2.4 in build.yml and 7.2.3 in libvmaf-build-matrix.yml, and the Level Zero loader existed at four versions at once. GitHub Actions cannot source a file at parse time, so run: steps source it at run time:

      - name: Fetch Level Zero loader source
        run: |
          set -a; . ./build-config.env; set +a
          git clone --depth 1 --branch "v${LEVEL_ZERO_VERSION}" \
            https://github.com/oneapi-src/level-zero.git /tmp/level-zero

For a value needed by a later step or by a with: block, use the loader, which writes KEY=value lines for every knob:

      - name: Load build config
        run: scripts/ci/load-build-config.sh >> "$GITHUB_ENV"

The development container's SDK stage copies build-config.env to /opt/vmafx/build-config.env and sources it in the Level Zero download RUN. Both the release tag and Debian package filename use LEVEL_ZERO_VERSION. There is no separate LEVEL_ZERO_VER argument: edit the shared setting and rebuild the development container to change the loader.

scripts/ci/check-workflow-versions.py rejects drifted literal Level Zero clone versions in workflows and verifies that the container downloads use the copied configuration. The Windows SYCL leg runs under cmd and mirrors the value by hand; the same check keeps that mirror aligned.

Renovate¶

Renovate's built-in dockerfile manager understands ARG X=image + FROM $X, so if it owned the Dockerfiles it would bump the ARG mirrors and leave build-config.env behind — failing the gate on Renovate's own PRs. Instead a single custom manager in renovate.json matches both the config line and the ARG form across every wired file, so one PR updates the shared pin and its mirrors together, and a packageRule disables the built-in manager on those files. docker/dev/*.Dockerfile keeps the built-in manager, because those pins are deliberately independent.

The Level Zero custom manager updates only LEVEL_ZERO_VERSION in build-config.env; its container and workflow consumers read that setting. ROCm uses the image manager's ROCM_BUILDER and ROCM_RUNTIME entries. The old ROCm manager for literal workflow versions no longer has an input and is removed. Renovate does not move ROCM_VERSION: a ROCm pull request fails scripts/ci/check-base-image-single-source.sh until the release is set there too, and its ROCm (hip runtime) rule labels it manual-review for the steps in ROCm and oneAPI bases.

scripts/ci/tests/test_renovate_file_patterns.py derives the wired files from the tree: every Dockerfile in the single-source gate's scope that declares an ARG default for an image key of build-config.env. A new mirror the custom manager does not select fails that test. docker/Dockerfile.tester was such a mirror until the ROCm 10.1.0 update: no manager selected its ROCM_BUILDER, so the update left it at 10.0.0.