CPU score drift — bisect triage (2026-05-02)¶

Status: investigation only, no fix. Severity: sub-golden-tolerance (places=2 = 0.005, drift = 1.07e-3). Not a CI blocker. Author: bisect run via /bisect-regression, 2026-05-02.

Symptom¶

Running the canonical Netflix normal pair through CPU vmaf_v0.6.1:

The drift is stable and reproducible; it falls inside the places=2 golden tolerance of 5e-3, so the golden gate does not fire. The testdata/netflix_benchmark_results.json snapshot is NOT a Netflix golden asset (per CLAUDE.md §9 it is a fork-added GPU/SIMD numerical snapshot).

Bisect result¶

The bisect found no first-bad commit on the fork's history. Reasoning:

1. The fork's master and the upstream v3.0.0 tag share no merge base — the fork was rooted from a different upstream snapshot in commit 41301496 ("feat(ai): T5-3e empirical PTQ accuracy across CPU/CUDA/OpenVINO EPs", #174, 2026-04-28). git merge-base v3.0.0 4cd3a8d8 returns no output.
2. The drift is present at every fork commit tested, including the fork's root commit 41301496. Bisect predicate (/tmp/bisect-predicate.sh) returns bad at 41301496^{commit}.
3. Therefore no fork-internal commit introduced the drift — it was inherited at the moment the fork was rooted.

Root cause¶

Per-feature score comparison (system v3.0.0 binary vs fork master tip CLI, identical inputs, identical model JSON, single thread, --cpumask 0 to force the C scalar path):

The drift is exclusive to integer_motion / integer_motion2. ADM and VIF are bit-identical.

The drift survives --cpumask 0 (force scalar) and an enable_asm=false build, ruling out SIMD divergence.

The smoking-gun diff is in the edge-mirror helpers used by motion's convolution stages. Compare core/src/feature/integer_motion.h:

v3.0.0 (off-by-one mirror, the buggy upstream behaviour the system binary was built with):

// j_tap >= width
j_tap = width - (j_tap - width + 1);
// i_tap >= height
i_tap = height - (i_tap - height + 1);

Fork master (the bug-fixed mirror):

j_tap = width - (j_tap - width + 2);
i_tap = height - (i_tap - height + 2);

The same +1 -> +2 fix is in the edge_8 helper in integer_motion.c. This correction is the upstream Netflix commit a44e5e611edf3bf02abe8cc7427ce30186396925 by Kyle Swanson (2026-04-17), titled "libvmaf/feature: port motion updates, bugfix for edge mirroring". The fork imported that commit at root.

The delta therefore is not a regression — it is the upstream bug fix producing slightly different (and now correct) edge taps for the motion convolution. The system v3.0.0 binary embeds the unfixed buggy edge-mirror; the fork, the snapshot's bench_all.sh sources, and any rebuild of the fork all carry the corrected version.

Snapshot consistency¶

The snapshot testdata/netflix_benchmark_results.json was authored at the fork root commit 41301496. The committed pooled value 76.668904 matches the system-installed /usr/local/bin/vmaf v3.0.0 binary (which was built from upstream Netflix v3.0.0, before the edge-mirror fix). Looking at bench_all.sh the bench runner's default VMAF path is /usr/local/bin/vmaf. So the snapshot was generated by the upstream-buggy binary, not by a build of the fork itself.

This means the snapshot has been stale vs every fork build that has ever existed. The 1.07e-3 drift is the constant offset between "snapshot-author-binary" (buggy mirror) and "fork-builds-itself" (fixed mirror).

Recommendation¶

Two viable next steps; both are out of scope for this triage PR.

Option A (recommended) — regenerate the snapshot¶

Run /regen-snapshots to rebuild testdata/netflix_benchmark_results.json against the fork's current CPU build. Justification in the regen commit: "snapshot was authored against the upstream v3.0.0 system binary which lacks the a44e5e61 edge-mirror fix; regenerating to match the fork's actual behaviour. Drift is +1.07e-3 on vmaf_v0.6.1 Netflix-normal, well inside the golden places=2 tolerance."

This brings snapshot and code into agreement and avoids future false alarms when /run-netflix-bench flags the same constant offset.

Option B — pin the bench runner to a fork-built binary¶

Update bench_all.sh to default VMAF to a freshly-built build/tools/vmaf rather than /usr/local/bin/vmaf. This eliminates the "snapshot authored by a different binary than fork builds produce" foot-gun permanently. (Would still require a one-time snapshot regen; pairs naturally with Option A.)

Why not "fix the code"¶

The fork-side +2 edge mirror is the correct behaviour, ported from Netflix upstream and now present in the fork's history at root. Reverting to the upstream-buggy +1 would re-introduce the upstream bug. Per CLAUDE.md "never weaken a test to make it pass" and "Netflix golden assertions are the correctness ground truth" — the golden tests (which use places=2) still pass on the fork build, so the implementation is correct under the documented gate.

Methodology notes / caveats¶

• Initial bisect run aborted with "no binary search possible" because every intermediate commit tripped a meson-setup error: older commits' enable_vulkan is a feature option (enabled/disabled/auto), not a boolean. Predicate fixed and bisect rerun cleanly.
• Bisect "first bad" landed on 41301496 only because the search tree treats the fork root as the earliest known commit. That hash should be read as "drift was present at every fork commit" — it is not a regression introducer in any meaningful sense.
• Predicate: build CPU-only, run vmaf on the Netflix normal pair via the --model path=... flag with --threads 1, parse pooled_metrics.vmaf.mean, good if |score - 76.668904| < 1e-4, bad if > 5e-4, skip otherwise. Predicate file: /tmp/bisect-predicate.sh (not committed).

Reproducer¶

# 1. System v3.0.0 (snapshot-baseline binary)
/usr/local/bin/vmaf \
  --reference  python/test/resource/yuv/src01_hrc00_576x324.yuv \
  --distorted  python/test/resource/yuv/src01_hrc01_576x324.yuv \
  --width 576 --height 324 --pixel_format 420 --bitdepth 8 \
  --model path=model/vmaf_v0.6.1.json --threads 1 \
  --output /tmp/sys.json --json -q
python3 -c "import json; print(json.load(open('/tmp/sys.json'))['pooled_metrics']['vmaf']['mean'])"
# -> 76.668905

# 2. Fork master tip (whichever build dir you have)
build/tools/vmaf <same flags> --no_cuda --no_sycl --no_vulkan
# -> 76.66783

# 3. Per-feature breakdown — the drift is fully attributable to integer_motion / integer_motion2

References¶

• Upstream Netflix commit a44e5e61 — "libvmaf/feature: port motion updates, bugfix for edge mirroring", Kyle Swanson, 2026-04-17.
• CLAUDE.md §8 (Netflix golden gate, places=2 tolerance).
• CLAUDE.md §9 (snapshot regeneration).
• bench_all.sh (testdata/bench_all.sh) — defaults VMAF to /usr/local/bin/vmaf if VMAF_BIN is unset.