Netflix benchmark baselines — how to reproduce them¶

To re-run the Netflix benchmark suite, follow Running the suite and compare against the recorded snapshot with Diffing a fresh run. This page documents the harness behind testdata/netflix_benchmark_results.json (the fork's recorded per-backend score and throughput snapshot for the three Netflix fixtures) and, at the end, what the last re-run measured.

Which harness writes which file¶

There are six benchmark scripts under testdata/, and the first three are easy to confuse:

testdata/compare_combined.py compares scores_cpu_*.json against scores_sycl_a380_*.json. It does not read netflix_benchmark_results.json, so it cannot be used to diff a fresh benchmark_netflix.py run against the recorded snapshot — compare the JSON directly (see Diffing a fresh run).

Prerequisites¶

benchmark_netflix.py drives an FFmpeg binary that links the fork's libvmaf and carries all three filters. The supported way to get one is the vmaf-dev-mcp container (dev-mcp.md), which ships FFmpeg with libvmaf, libvmaf_cuda and libvmaf_sycl already wired up.

The YUV fixtures live in python/test/resource/yuv/ and are gitignored, so they exist only in a full checkout. Point VMAF_YUVDIR at that checkout when running from a worktree.

Two environment variables select host-specific inputs (both defaults are host-specific and are expected to be set):

Running the suite¶

To measure current libvmaf rather than the one baked into the image, build the library out-of-source inside the container and put it ahead of the installed one on LD_LIBRARY_PATH. FFmpeg resolves libvmaf.so.3 at load time, so no FFmpeg rebuild is needed as long as the public headers under core/include/libvmaf/ have not changed shape.

1. Build current master's libvmaf inside the container, out of source.

   docker exec vmaf-dev-mcp bash -lc '
     set +u; source /opt/intel/oneapi/setvars.sh >/dev/null 2>&1; set -u
     CC=icx CXX=icpx meson setup /tmp/bench-build /workspace/core \
         -Denable_cuda=true -Denable_sycl=true \
         -Denable_hip=false -Denable_metal=disabled -Denable_dnn=disabled \
         -Denable_mcp=false -Denable_tests=false \
         --buildtype=release -Db_lto=false -Dc_args=-march=native
     nice -n 10 ninja -C /tmp/bench-build -j 4'
   

2. Run the suite against it.

   docker exec vmaf-dev-mcp bash -lc '
     set +u; source /opt/intel/oneapi/setvars.sh >/dev/null 2>&1; set -u
     export LD_LIBRARY_PATH=/tmp/bench-build/src:$LD_LIBRARY_PATH
     export VMAF_FFMPEG=/usr/local/bin/ffmpeg
     export VMAF_YUVDIR=/workspace/python/test/resource/yuv
     export VMAF_SYCL_RENDER_NODE=$(ls /dev/dri/renderD*)   # see below
     mkdir -p /tmp/benchrun && cp /workspace/testdata/benchmark_netflix.py /tmp/benchrun/
     cd /tmp/benchrun && python3 benchmark_netflix.py'
   

The script writes its output JSON next to itself, so running the copy in /tmp/benchrun keeps the tracked snapshot untouched. That is deliberate: overwriting testdata/netflix_benchmark_results.json requires the /regen-snapshots justification.

Picking the SYCL render node¶

benchmark_netflix.py imports frames onto the Intel GPU through QSV/VA-API, so it needs the render node that the iHD driver claims. Node numbering is not stable across hosts or across PCI re-enumeration, so find the right node before running:

docker exec vmaf-dev-mcp bash -lc \
  'for n in /dev/dri/renderD*; do echo "== $n"; vainfo --display drm --device $n 2>&1 | grep -m1 "Driver version"; done'

Pick the node whose driver line reads Intel iHD driver, and pass it as VMAF_SYCL_RENDER_NODE. The script reads only that environment variable (testdata/benchmark_netflix.py); it has no command-line option for it.

Diffing a fresh run¶

python3 - <<'PY'
import json
rec = json.load(open("testdata/netflix_benchmark_results.json"))
new = json.load(open("/tmp/benchrun/netflix_benchmark_results.json"))
for fixture in rec:
    for backend in rec[fixture]:
        r, n = rec[fixture][backend], new.get(fixture, {}).get(backend)
        if not n or "frames" not in n:
            print(f"{fixture:22s} {backend:5s} MISSING in fresh run"); continue
        md = max(abs(a - b) for a, b in zip(r["frames"], n["frames"]))
        print(f"{fixture:22s} {backend:5s} pooled {r['pooled']:.6f} -> {n['pooled']:.6f} "
              f"(delta {n['pooled']-r['pooled']:+.2e}, max per-frame {md:.2e})")
PY

Any non-zero score delta is a correctness finding: open a docs/state.md row, do not regenerate the snapshot. A throughput delta is a performance observation and never justifies a rewrite on its own.

History¶

Re-run of 2026-09-06, commit cd52f2670¶

This is a dated measurement, kept as the record of how far the snapshot had drifted. Toolchain versions have moved since (see build-config.env).

Host and method¶

Every figure below is from a command run on that host. Timing figures are medians of 5 repetitions with the min/max spread quoted, and are not a substitute for a quiet-host baseline.

Reading the harness's own PASS/DIFF column¶

benchmark_netflix.py prints each pooled score against the Netflix golden value of the pair (src01_576x324 = 76.66783025, the vmaf CLI assertion in python/test/vmafexec_test.py) and tags it OK within 5e-5. The CPU row agrees with it to about 3e-6. Earlier versions of the harness compared against 76.66890519623612, the value the Python harness asserts in python/test/quality_runner_test.py to two places, so the CPU row always read DIFF; that was a wrong reference, not drift. Compare against the snapshot when asking whether master has drifted.

Scores versus the recorded snapshot¶

CUDA is quoted at its modal value (see the blockers table below).

None of these deltas come from the 2026-09-06 GPU merges:

• Rebuild result. Rebuilding 5a080300e (the commit immediately before #1307, #1312 and #1324) with the same flags gives CUDA 76.667830 and SYCL 76.667745 on the 576x324 pair, the same drift versus the recorded snapshot.
• CPU delta. The three merges moved CPU per-frame values by up to about 8e-6; the pooled value is unchanged at six decimals.
• Key-set change. The SYCL frames[].metrics key set shrank from 35 keys to 24. The CUDA key set is 14 on both commits.
• Age. The recorded snapshot dates from PR #309 (2026-05-02), so the drift accumulated across four months.

Throughput (FFmpeg filter path, 576x324 48f and 1080p 3f)¶

Whole-process wall time including FFmpeg start-up, matching how benchmark_netflix.py computes fps. Median of 5, load average 9.6-9.8 during the sweep. These numbers are observations, not a recorded baseline: no throughput baseline is written while the CUDA path was non-deterministic (ADR-1192).

The CPU-beats-GPU ordering at these sizes is expected: 48 frames of 576x324 and 3 frames of 1080p do not amortise upload and launch cost. See benchmarks.md for the 4K numbers where CUDA dominates.

Blockers found by this run¶

Two GPU defects were reproduced. Both were confirmed present on 5a080300e as well, so they were pre-existing rather than introduced on 2026-09-06. Both are closed now (see the status note at the top of this page).

--threads N abort. --gpumask=0 (CUDA) or --sycl_device=0 (SYCL) combined with any --threads value emitted feature "VMAF_integer_feature_motion2_score" cannot be overwritten at index N for every frame, then libvmaf ERROR context could not be synchronized / problem flushing context, and exited 234 with no output file. Dropping --threads made both backends succeed and score correctly (CUDA 76.667830, SYCL 76.667746 on the 576x324 pair, 10/10 runs each).

testdata/bench_all.sh hard-codes --threads 1, so every CUDA and SYCL row it printed on this host before this run was a masked failure, reported as SKIP (... backend likely unavailable) because the script also discarded stderr. It now keeps stderr in $VMAF_BENCH_OUTDIR/<row>.err and prints FAIL (vmaf exited 234: problem flushing context; see ...). Its flag sets also dropped --no_vulkan, which current CLI builds reject as unrecognized (ADR-0726 removed the Vulkan backend); the flag was ignored rather than fatal, so it had gone unnoticed.

Non-deterministic libvmaf_cuda filter. On the 576x324 48-frame pair, 10 of 40 runs on cd52f2670 and 8 of 40 runs on 5a080300e returned a pooled score other than 76.667830. The bad runs differ in one or two individual frames (for example frame 1 = 0.0, or frame 3 = 50.834 where CPU says 81.925763), not in a global offset. CPU (10/10) and SYCL (10/10) through the same FFmpeg build are bit-stable, and CUDA through the vmaf CLI without a thread pool is bit-stable (10/10). The two failure rates are within binomial noise of each other, so the merges neither caused nor worsened it.