dev-MCP Docker Container¶

The dev-MCP container runs the full VMAF fork inside Docker with the three active Linux GPU backends enabled (CUDA, SYCL, HIP), the CPU backend, and the production Go MCP stdio server (vmafx-mcp). Metal remains a macOS-only backend. It is the standard environment for:

• Live probing of VMAF scores across all backends from a single shell.
• Running the continuous smoke-probe cron (smoke-probe-cron service).
• Reproducing build regressions on GPU paths other than the host's primary GPU (for example: catching HIP toolchain regressions on an NVIDIA-only host).

The design decision is recorded in ADR-0451. The agent rule that makes this container the default for vmaf, vmaf-tune, ai and MCP work is hard rule 12 of agent-hard-rules.md.

Prerequisites¶

Required¶

Optional¶

How to start¶

Build (if needed) and start the container with the wrapper from the repository root:

# CPU only (no GPU passthrough)
./dev/scripts/dev-mcp-up.sh

# With NVIDIA GPU passthrough
NVIDIA_VISIBLE_DEVICES=all CONTAINER_RUNTIME=nvidia \
    ./dev/scripts/dev-mcp-up.sh

The first build downloads all GPU SDK layers and compiles libvmaf from source. Expect 20-40 minutes on a typical workstation; later builds use the layer cache and take 1-3 minutes when only Python packages change. To rebuild without starting, see Building and maintaining the image.

The wrapper starts two services:

1. vmaf-dev-mcp - the primary container. It runs vmafx-mcp through docker exec -i stdio when requested. The service healthcheck verifies vmaf --version and, when /dev/nvidia0 is exposed, requires a successful nvidia-smi driver query. It is not a socket check. A 45-second start period covers CUDA driver cold-start before dependent services are admitted.
2. vmaf-smoke-probe-cron - waits for the primary to be healthy, then probes every 15 minutes.

Both services write probe files to .workingdir/dev-mcp-probes/ on the host.

Host video and render groups¶

Group IDs differ across distributions, so the container must join the host's video and render groups. Export HOST_GID_VIDEO and HOST_GID_RENDER before starting; the wrapper reads them automatically. You can also pass them inline:

HOST_GID_VIDEO=$(getent group video | cut -d: -f3) \
HOST_GID_RENDER=$(getent group render | cut -d: -f3) \
CONTAINER_RUNTIME=nvidia \
docker compose -f dev/docker-compose.yml --project-directory . up -d dev-mcp

The defaults baked into docker-compose.yml (44 for video, 109 for render) match common Ubuntu installations. Override them whenever getent group video returns a different GID (for example, Arch Linux uses 985/986).

How to attach¶

# Interactive bash shell inside the running dev-mcp container
./dev/scripts/dev-mcp-shell.sh

# Run a specific command
./dev/scripts/dev-mcp-shell.sh vmaf-dev-mcp vmaf --version
./dev/scripts/dev-mcp-shell.sh vmaf-dev-mcp vmaf --help

One-shot commands¶

Started without a command, the image stays up so you can docker exec into it. Started with a command, it runs that command and exits with its status:

# A throwaway probe: runs, prints, exits, and the container is removed
docker run --rm vmaf-dev-mcp:local pkg-config --modversion vpl

# The same against the running dev-mcp container
docker exec vmaf-dev-mcp clang-tidy --version

Use docker run --rm (or docker exec vmaf-dev-mcp, agent hard rule 12) for one-shot work. Before this behaviour every one-shot docker run left a running container with its healthcheck, because the entrypoint ignored its arguments.

Inside the container the full environment is initialised:

• vmaf CLI - /usr/local/bin/vmaf
• vmafx-mcp - /usr/local/bin/vmafx-mcp (Go; ADR-1229 replaced the Python mcp-server/vmaf-mcp package, which is deprecated)
• GPU SDKs - nvcc, icpx, hipcc in PATH
• testdata - /workspace/testdata/ (read-only bind mount from host repo)
• models - /workspace/model/ (read-only)

To list the registered feature extractors, call the MCP tool list_extractors (the manual probe below does this for you).

How to manually probe¶

Run a single smoke probe outside the cron cycle:

./dev/scripts/dev-mcp-probe.sh

This executes smoke-probe-loop.sh --once inside the running container and writes probe-<timestamp>.json to .workingdir/dev-mcp-probes/. If jq is installed on the host the result is pretty-printed to stdout.

How to stop¶

# Stop, keep volumes (probe history preserved)
./dev/scripts/dev-mcp-down.sh

# Stop and remove volumes (clears socket volume; probe bind-mount preserved)
./dev/scripts/dev-mcp-down.sh --volumes

How to interpret probe outputs¶

Each probe file follows this schema:

{
  "ts": "2026-05-15T14:30:00Z",
  "host_id": "myhostname:abc123def456",
  "backend_results": {
    "cpu":  { "score": 94.32301, "duration_ms": 3200, "error": null },
    "cuda": { "score": 94.32301, "duration_ms":  820, "error": null },
    "sycl":   { "score": null,  "duration_ms":    0, "error": "ENOSYS: no SYCL device" },
    "hip":    { "score": null,  "duration_ms":    0, "error": "ENOSYS: no HIP device" }
  },
  "mcp_results": {
    "list_features": { "feature_count": 14, "duration_ms": 45, "error": null },
    "compute_vmaf":  { "score": 94.32301, "duration_ms": 3250, "error": null }
  }
}

The list_features and compute_vmaf names in the probe JSON are stable schema keys retained for existing probe consumers. The operations behind them are the current Go MCP tools list_extractors and vmaf_score, respectively. Both MCP probes and every backend probe score the same pair with the same model, so compute_vmaf carries the CPU score.

Each MCP probe performs the required initialize / notifications/initialized handshake before its tools/call request and keeps the server's stdin open until the matching response arrives. Closing stdin after writing the request disconnects the Go SDK session and can race the asynchronous response.

Expected values¶

• CPU score: 94.32301 for the committed 48-frame pair with vmaf_v0.6.1 (checked against a CPU build of master).
• CUDA / SYCL / HIP scores: equal to the CPU's bits for every extractor declared an exact twin (one file per twin in scripts/ci/exact_twins.d/, listed in the generated table). A feature that is not declared exact stays inside the tolerance of the cross-backend gate (for example ciede at 1e-9); see cross-backend-gate.md and ADR-0214.
• SYCL: ENOSYS on hosts without Intel GPU or oneAPI runtime; normal.
• HIP: error on NVIDIA-only hosts; normal.

Common error patterns¶

Known limitations¶

For the vmaf-tune report row: after a rebuild, vmaf-tune report --format both produces a self-contained HTML and Markdown report with inline charts. If you see ModuleNotFoundError: No module named 'matplotlib' inside the container, the image predates the ADR-0498 commit (2026-05-18); rebuild it with docker compose -f dev/docker-compose.yml build dev-mcp.

Backend matrix¶

On a host with NVIDIA, Intel Arc and AMD silicon and the NVIDIA Container Toolkit installed, every libvmaf backend should run inside the container:

Reproducer:

docker exec vmaf-dev-mcp bash -c '
  for B in cpu cuda sycl hip; do
    vmaf --reference /workspace/python/test/resource/yuv/src01_hrc00_576x324.yuv \
         --distorted /workspace/python/test/resource/yuv/src01_hrc01_576x324.yuv \
         --width 576 --height 324 --pixel_format 420 --bitdepth 8 \
         --backend $B --json --output /tmp/probe_$B.json
    echo "rc=$? backend=$B"
  done
'

Environment-variable contract¶

The container pins the ROCm variables at compose-up time and carries the oneAPI runtime libraries needed for SYCL discovery. The VMAFx Vulkan backend was removed by ADR-0726 and is not part of this contract.

Operators on hosts with a ROCm-supported GPU on the allowlist (gfx1030 / gfx1100 / gfx1101 desktop / workstation parts) can override HSA_OVERRIDE_GFX_VERSION to the empty string at docker compose up time to remove the lie.

FFmpeg encoder matrix¶

The in-image FFmpeg is built with the fork's full encoder set so that vmaf-tune compare sweeps can address every codec the project supports without skipping rows with hardware encoder not available: ... not compiled into ffmpeg (ADR-0543). The matrix:

To verify the in-image listing after a rebuild:

docker exec vmaf-dev-mcp ffmpeg -hide_banner -encoders 2>&1 \
    | grep -E "libsvtav1|libvvenc|libvpx-vp9|nvenc|qsv|amf|vpl" \
    | head -20

Expected (assuming the build-time encoder probe in stage 3.5 logged no WARN ... missing):

 V....D libsvtav1            SVT-AV1(Scalable Video Technology for AV1) encoder
 V..... libvvenc             libvvenc-based VVC encoder
 V....D libvpx-vp9           libvpx VP9
 V....D h264_nvenc           NVIDIA NVENC H.264 encoder
 V....D hevc_nvenc           NVIDIA NVENC hevc encoder
 V....D av1_nvenc            NVIDIA NVENC av1 encoder
 V....D h264_qsv             H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10 (Intel Quick Sync Video acceleration)
 V....D hevc_qsv             HEVC (Intel Quick Sync Video acceleration)
 V....D av1_qsv              AV1 (Intel Quick Sync Video acceleration)
 V....D h264_amf             AMD AMF H.264 Encoder
 V....D hevc_amf             AMD AMF HEVC encoder
 V....D av1_amf              AMD AMF AV1 encoder

Hardware-encoder runtime failure modes¶

Each encoder has two checks. "Compile-in" asks whether the binary advertises the encoder. "Runtime-ok" asks whether a 1-frame dummy encode succeeds. vmaf-tune compare runs both in compare.py::probe_encoder_available. The container locks down the compile-in promise; runtime failures produce stable row-level skip strings:

Reproducer: full cross-codec compare sweep¶

docker exec vmaf-dev-mcp bash -c '
  cd /workspace && PYTHONPATH=/workspace/tools/vmaf-tune/src:$PYTHONPATH \
  python -c "from vmaftune.cli import main; raise SystemExit(main())" compare \
    --src /workspace/.corpus/bbb_e2e/bbb_sunflower_1080p_60fps_normal.mp4 \
    --width 1920 --height 1080 --framerate 60 \
    --target-vmafs 85,90,92,95 \
    --encoders libx264,libx265,libsvtav1,libaom-av1,libvvenc,libvpx-vp9,h264_nvenc,hevc_nvenc,av1_nvenc,h264_qsv,hevc_qsv,av1_qsv,h264_amf,hevc_amf,av1_amf \
    --duration 5 --sample-clip-seconds 3 --max-iterations 3 \
    --score-backend cuda --format json --output /tmp/v11_1080p_cmp_full.json'

Encoders that are not runtime-available on the host produce per-row ok=false entries with the diagnostic strings above; the sweep does not abort.

Building and maintaining the image¶

This section is for anyone rebuilding the container or changing dev/Containerfile. Day-to-day use needs only the sections above.

Rebuild the image¶

Use the wrapper from the repository root:

./dev/scripts/container-build.sh

Prefer it over a bare docker compose build. Before spending 20-40 minutes it checks what it is about to build from, and it records the answer in the image (ADR-1195). The bare compose form still works and is what the wrapper calls underneath:

docker compose --project-directory "$(pwd)" -f dev/docker-compose.yml build

It performs no source check and leaves the image recording source_rev=unknown.

Which source is in the image?¶

A rebuild only picks up work that is in the checkout you build from. If the checkout is behind master, the build still succeeds and the image is still newer than every commit in the repository, but it does not contain the commits you rebuilt for. Timestamps cannot tell the two cases apart, so ask the image directly (the incident that motivated this check is under History):

# Is this checkout a valid build context right now?
bash scripts/dev/check-container-source.sh --pre-build

# What was an existing image actually built from?
bash scripts/dev/check-container-source.sh --image vmaf-dev-mcp:local

Exit codes:

Treat 2 as "rebuild before trusting anything measured in here", not as a pass. Before citing a number produced inside the container (a GPU smoke, a benchmark, a parity sweep), check the image first: an unattributable measurement is worse than none, because it looks like evidence.

Optional GitHub API authentication¶

The Intel NEO resolver works without credentials. On a shared network, an authenticated GitHub API request can avoid the lower anonymous rate limit. For Compose and the wrapper scripts, export GITHUB_TOKEN before the build; dev/docker-compose.yml maps it to the optional github_token build secret. An unset or empty value keeps the build anonymous.

For a raw anonymous build, omit the secret entirely:

env -u GITHUB_TOKEN docker build \
  --file dev/Containerfile \
  --target libvmaf-build \
  --tag vmaf-dev-mcp:local \
  .

For an authenticated raw build, pass the exported variable as a BuildKit secret:

docker build \
  --file dev/Containerfile \
  --target libvmaf-build \
  --tag vmaf-dev-mcp:local \
  --secret id=github_token,env=GITHUB_TOKEN \
  .

Never pass this credential with --build-arg. The secret is mounted only for the NEO metadata-fetch instruction and is not recorded in image layers, metadata, or provenance. See ADR-1271.

CI stage builds¶

CI builds stage images through one script, scripts/ci/build-dev-container-stage.sh <target> <image-tag>. It accepts two targets, uses no external layer cache and passes no build arguments:

The script no longer accepts build-deps: no CI job builds that stage on its own. Run it to reproduce either job's image:

bash scripts/ci/build-dev-container-stage.sh libvmaf-build vmaf-dev-mcp:local
bash scripts/ci/build-dev-container-stage.sh release-build vmafx-release-build:local

See publishing for the release build that runs inside release-build.

CI builds the image only up to libvmaf-build (ADR-0819), plus release-build on its own for the release rehearsal (ADR-1346). The final dev-mcp stage is therefore covered by a static contract instead (ADR-1343):

python3 scripts/ci/check-dev-container-stage-inputs.py

The check fails in two cases:

• A stage's RUN reads a pip requirement or constraint file under /build/vmaf/ that no COPY into that stage or its parent stages provides.
• A COPY source is missing from the repository.

It runs from pre-commit whenever dev/Containerfile, a lock file or the check itself changes. Fix a failure by adding the matching COPY to the stage that reads the file; copying it in the final stage keeps the libvmaf and FFmpeg layers cached.

Compiler selection¶

The stages use two GCC versions:

• In build-deps, gcc, g++, gcc-ar, gcc-nm and gcc-ranlib all point at GCC 13 through update-alternatives.
• A later stage installs Ubuntu's default GCC 15, which replaces those links, so libvmaf-build and the final image compile with GCC 15.
• release-build installs Debian's default gcc package, which provides gcc-ar and the LTO linker plugin itself.

The gcc-ar link matters in build-deps. Meson archives static libraries with gcc-ar when it exists; without it, plain ar cannot index GCC's LTO objects in that stage and every LTO link against libvmaf.a fails with undefined references.

Build failures and cache use¶

• Every stage that runs build pipelines explicitly enables Bash pipefail, so an upstream command failure cannot be hidden by a successful output filter.
• The golden-suite import and collection checks fail the build; collection failures print the captured pytest diagnostics.
• Hardware-availability probes keep their documented warning behaviour in a build sandbox without GPUs.
• The libvmaf configure and compile commands share the exported CCACHE_DIR backed by the BuildKit cache mount.
• Install steps address their build trees explicitly, and FFmpeg cleanup runs from outside the directory it removes.
• The Go artifact stage verifies seven outputs without parsing filenames as lines, then returns to the vmaf user. The final runtime also uses vmaf.

Run hadolint dev/Containerfile to check the Dockerfile and embedded shell before a rebuild. This static check does not establish native build or GPU runtime acceptance.

ONNX Runtime in the image¶

The container installs ONNX Runtime's native CPU archive for libvmaf's C/C++ API. That archive does not add CUDA or ROCm execution providers; libvmaf's own GPU feature backends and the Python onnxruntime package are separate components. A provider-enabled ORT installation needs the corresponding runtime libraries, as described in ADR-0113.

Host-kernel and container-userspace version pins¶

Intel NEO compute-runtime and ROCm KFD userspace are version-pinned through Containerfile ARGs to match the host kernel's i915 / xe / KFD ioctl ABI (ADR-0543). A mismatch silently degrades vmaf --backend sycl|hip to CPU.

The NEO resolver accepts only GitHub-hosted HTTPS URLs. API credentials are sent only to api.github.com and are removed before cross-host redirects. Metadata reads are bounded, package downloads use atomic temporary files with transient-error retries, and checksum or Debian-package validation failures remove the invalid output before the build stops.

dev-mcp-entrypoint.sh emits a runtime visibility probe on container start (ADR-0543):

• WARN: SYCL GPU NOT detected
• WARN: HIP HSA GPU agent NOT detected

Either warning means the host kernel has revved past the pinned userspace ABI. Bump the ARG and rebuild rather than working around the fallback. SYCL detection accepts anchored Level Zero and OpenCL GPU records; HIP detection accepts anchored Name: gfx... or Device Type: GPU records, never diagnostic prose that merely mentions a GPU token.

Where to find current versions:

• Latest NEO release tag: https://github.com/intel/compute-runtime/releases/latest
• ROCm release notes: https://rocm.docs.amd.com/en/latest/about/release-notes.html
• ROCm images: https://hub.docker.com/r/rocm/dev-ubuntu-26.04/tags

Do not consult https://repo.radeon.com/rocm/apt/ for the current version; that channel has been frozen at 7.2.4 since AMD moved to TheRock.

History¶

• 2026-09-06, stale checkout. The container was rebuilt specifically to pick up the GPU default-model fixes (#1307, #1312, #1324). The checkout was 28 commits behind, and the resulting image had none of them. It was noticed only because a test file was missing; a GPU smoke run instead would have reported green numbers for code that was not in the image. This is why scripts/dev/check-container-source.sh and the source_rev marker exist (ADR-1195).
• HSA_OVERRIDE_GFX_VERSION removal. The variable used to be 10.3.0. AMD gfx1036 (Raphael iGPU, RDNA2 IP rev 10.3.6) was not on the ROCm 6.x/7.x supported-GPU allowlist, so without the override hsa_init() returned HSA_STATUS_ERROR_OUT_OF_RESOURCES and rocminfo reported "Unable to open /dev/kfd read-write: Invalid argument" even with /dev/kfd bind-mounted. ROCm 10 supports gfx1036 natively, so ADR-1225 superseded ADR-0542 on this point.