Benchmarks¶

This page records fork-added benchmark results (GPU backends, SIMD paths, --precision overhead). Every table carries the date, host, commit and command of its measurement; tables that lack them are collected at the end under Unattributed history. Start with the headline table.

Current headline numbers¶

Upstream A/B (ADR-1228)¶

This is the only section that compares the fork against upstream Netflix/vmaf at a pinned commit. It answers whether the fork is worth using on throughput grounds, and whether it stayed exact while getting there. Every other table on this page compares the fork against itself.

CPU path only. Upstream has no SYCL, HIP or Metal backend and its CUDA backend covers a different feature set, so a GPU comparison would measure the hardware rather than the work. Per-backend numbers stay in backend-perf-baselines.md.

Harness¶

testdata/bench_upstream_ab.py builds upstream at the recorded parity head and this tree with the golden build profile, both through the upstream parity guard (ADR-1487). It then runs both binaries over the same fixtures with the ADR-1185 discipline:

• one discarded warmup,
• --runs timed repetitions reported as the median,
• spread and load average recorded.

The score verdict is the guard's: the model's values at %.17g on the same fixtures against the allowlist of recorded deviations. The "Score delta" column below is the six-decimal pooled difference of the two tools and is informational.

testdata/bench_upstream_ab.py --runs 5 --json /tmp/ab.json
testdata/bench_upstream_ab.py --upstream-ref v3.2.0   # another upstream commit or tag

To check score equality with upstream, run the guard (make upstream-parity, see upstream parity guard); the harness has no score-delta flag.

2026-09-07 — fork 257ac1ec4 vs upstream v3.2.0, ryzen-4090-arc, single-threaded¶

Measured with the six-decimal ceiling the harness had before the guard.

Reading the result¶

On the inherited path (the vmaf_v0.6.1 model, whose four features are integer_adm, integer_vif, integer_motion and integer_aim) the fork is at parity with upstream.

• What is equal: the fork has not regressed the code it inherited. That is the right result to want there.
• What it does not say: nothing about the fork's speed in general. Upstream already ships AVX2 and AVX-512 for exactly those four features, so picking that model picks the one workload where both projects are equally optimised.
• Both statements are true: the fork has not made the inherited integer path faster, and it is substantially faster across the surface it added SIMD to. A single geomean over one model would have hidden both facts, which is why the next section reports per-feature rows.

2026-10-02 — score delta closed (ADR-1475)¶

The score deltas of the 2026-09-07 table were one expression: the quantisation step of integer ADM formed its exponent in double where Netflix forms it in float (ADR-1475). With Netflix's expression restored, the score comparison passes against Netflix master cea2b4d8 on all four fixtures, the 200-frame 3840x2160 clip included. Before the fix the delta was 4.0e-06 on the 1 px checkerboard pair and 1.0e-06 on the 4K clip. No timing was recorded in that run. The tracking row T-UPSTREAM-AB-SCORE-DELTA-2026-09-07 in state.md is closed.

Where the fork actually wins: features upstream left scalar¶

Measured in the 2026-09-07 session of the table above (fork 257ac1ec4 against upstream v3.2.0, ryzen-4090-arc, per ADR-1228): the same 48-frame 1920x1080 pair, single-threaded, --feature X --no_prediction. The fork adds x86 SIMD sources upstream does not have.

† startup-dominated at 24 ms; not a real measurement.

float_ms_ssim is the headline: upstream has no ms_ssim_decimate SIMD at all, so it runs the decimation scalar. psnr_hvs and float_ssim are the same story at smaller scale. The rows near 1.00x are where upstream is already vectorised or where the kernel is memory-bound.

Features that exist only in the fork (for example speed_qa, delta_e_itp, niqe, brisque, pu21, ssimulacra2 and the tiny-AI extractors) have no upstream counterpart to compare against and are therefore absent from both tables.

Hardware profiles¶

The ryzen-4090-arc profile is the canonical fork bench host: a single machine that exposes CUDA (RTX 4090), SYCL (Arc A380 via oneAPI Level Zero) and HIP, so all active backends can run back-to-back from one shell. The xeon-arc and m4-pro profiles are defined for future runs; no table on this page uses them. The Vulkan backend was removed per ADR-0726; historical Vulkan bench rows are preserved in the history sections for reference.

Refreshed per-backend baselines (2026-09-06, ryzen-4090-arc)¶

Produced by testdata/bench_backends.py on commit cd52f2670; methodology in ADR-1185, reproduce steps in docs/development/backend-perf-baselines.md. Every cell is the median of 3 timed runs after one discarded warmup, --threads 1, one exclusive backend per run.

model/vmaf_v0.6.1.json (the model every historical row below used)¶

Default model — no --model flag (vmaf_v1.0.16_3d0h, ADR-1169)¶

This is what a caller who names no model actually pays.

What the default model costs¶

The comparison the retrain planning asked for, as a ratio of the two tables above (same fixture, same backend, same session):

Reading the ratios¶

Three things fall out of this, and only the first is comfortable:

1. On CPU the default model is not uniformly more expensive. It costs 38 % more per frame on the 576×324 pair but is faster on 1080p and 4K content. The v1 feature set is not simply "v0.6.1 plus more work" — it trades a different mix, and the mix's cost is resolution-dependent.

2. On CUDA the default model is dramatically worse, and worse the bigger the frame gets. It falls to 0.05× at 4K, where it is roughly half the speed of the CPU running the same model (9.00 fps vs 17.52 fps).

   During that run the process sat at ~97 % CPU with the RTX 4090 at ~34 % utilisation. This is the signature of ADR-1183's twin gating: features whose GPU twin does not support the model's options are dispatched to the CPU, so the run pays the host↔device transfer cost and then does the work on the host anyway.

   SYCL and HIP had the same class of gap in their ADM twins until ADR-1362 and ADR-1525 gave them their own AIM passes.

3. GPU offload only pays at 4K, and only for v0.6.1. Every 1080p×3f cell has CUDA losing to CPU by 2.5–5×, which is expected — three frames cannot amortise context creation — but the 4K default-model row shows the loss persisting into a workload that should be firmly GPU-favourable.

The retrain planning number¶

Stated plainly: on this host the default model costs 1.6× the CPU time of v0.6.1 on the 576×324 golden pair, and on CUDA it gives up the entire GPU speedup — 4K throughput falls from 167.16 fps to 9.00 fps, well below the CPU's own 17.52 fps.

How to reproduce¶

The commands below reproduce the dated tables of the history sections; the baselines above come from testdata/bench_backends.py (see backend-perf-baselines).

1. Build with all backends, with oneAPI sourced for icx/icpx and Arc visibility.

   source /opt/intel/oneapi/setvars.sh
   CC=icx CXX=icpx meson setup core/build core \
       -Denable_cuda=true -Denable_sycl=true \
       -Db_lto=false --buildtype=release
   ninja -C core/build
   

2. Acquire the fixtures (gitignored; do not commit them). The Netflix golden 576x324 and 1080p_5frames pairs already live in python/test/resource/yuv/ in the main checkout. For the BBB 4K 200-frame pair, download the source from archive.org and encode the reference.

   mkdir -p testdata/bbb
   curl -L https://archive.org/download/big-buck-bunny-4k-60fps/BigBuckBunny4k60fps.mp4 \
       -o /tmp/bbb4k.mp4
   ffmpeg -y -i /tmp/bbb4k.mp4 -frames:v 200 -pix_fmt yuv420p -s 3840x2160 \
       testdata/bbb/ref_3840x2160_200f.yuv
   

3. Encode the distorted 4K clip (libx264 CRF 35 round trip). Any equivalent round trip works; the absolute pool drifts with codec parameters but the fps numbers do not.

   ffmpeg -y -i /tmp/bbb4k.mp4 -frames:v 200 -c:v libx264 -crf 35 -preset veryfast \
       -pix_fmt yuv420p -s 3840x2160 -f rawvideo - \
       | ffmpeg -y -f rawvideo -pix_fmt yuv420p -s 3840x2160 -i - \
           -pix_fmt yuv420p -s 3840x2160 testdata/bbb/dis_3840x2160_200f.yuv
   

4. Run the bench.

   VMAF_BIN="$(pwd)/core/build/tools/vmaf" bash testdata/bench_all.sh
   

5. Verify that each backend engaged, from the per-row emitted key count, the pool, the throughput and stderr. A GPU key count equal to the CPU's is a fallback warning; never compare against a frozen expected count.

For the SIMD breakdown and --precision overhead numbers, see the harness scripts under testdata/ (or the inlined repeat_bench.py / simd_bench.py / precision_bench.py from the T7-37 PR description).

Quick CLI smoke benchmark¶

testdata/bench_quick.py runs the CPU CLI three times for every complete ref_<width>x<height>_48f.yuv / dis_<width>x<height>_48f.yuv pair it finds. It reports best and average fps without writing a snapshot:

VMAF_BIN="$(pwd)/core/build/tools/vmaf" \
VMAF_TESTDATA="$(pwd)/testdata" \
python3 testdata/bench_quick.py

Each Git lookup is limited to 10 seconds and each VMAF run to 300 seconds. The command exits 1 on a VMAF failure or timeout, exits 2 when no complete fixture pair exists, and prints the captured diagnostic on stderr. Missing resolutions are skipped; a reference without its matching distorted input is not treated as a runnable pair.

FFmpeg lavfi performance harness¶

testdata/bench_perf.py runs the FFmpeg filter path used by the historical perf_benchmark_results.json snapshot. It is useful when the question is "how fast is FFmpeg decode/upload/filter end-to-end?" rather than "how fast is the vmaf CLI binary?"

The harness is portable across checkouts:

python3 testdata/bench_perf.py \
    --ffmpeg /path/to/ffmpeg \
    --backend cpu \
    --backend cuda \
    --runs 3

Environment overrides are also supported:

The committed raw 4K BBB pair remains the required fixture. The 1080p raw pair and MP4 decode test are optional by default; pass --require-all when you want a strict lab run that fails on any missing configured input. Use --list-tests to audit fixture availability and --dry-run to print the exact FFmpeg commands without touching hardware.

History¶

Older runs, retained for reference. Do not use them as current numbers.

Historical backend comparison (Netflix normal pair, 576x324, 48 frames)¶

Source: python/test/resource/yuv/src01_hrc00_576x324.yuv vs …hrc01…. Model: model/vmaf_v0.6.1.json. Threads: 1. Precision: CLI default %.6f per ADR-0119. Numbers averaged over 5 wall-clock reps after 1 warmup; standard deviation in parentheses. Commit 41301496 on ryzen-4090-arc.

Key-count check for this dated run. At commit 41301496, the emitted sets were CPU=15 (including integer_aim/integer_motion3/integer_adm3), CUDA=12, and SYCL=34 (including raw _num/_den intermediates). Those values document this run; they are not permanent backend contracts. On 2026-09-06 at cd52f2670, the FFmpeg filter path instead emitted CPU=15, CUDA=14, and SYCL=24. The harness records actual counts and flags equality with CPU as a fallback signal to corroborate with pool and throughput. See core/AGENTS.md §"Backend-engagement foot-guns".

Unattributed history (do not cite)¶

The tables below carry no date, commit, host profile or command of their own. They are kept because they show the shape of the results, not as measurements to quote. Where a source is stated, it is the assumption the original text made.

Backend comparison (1080p, 5 frames)¶

Source: python/test/resource/yuv/src01_hrc{00,01}_1920x1080_5frames.yuv. Same setup as 576×324.

CPU outpaces CUDA at 1080p × 5 frames because dispatch overhead dominates the workload — only 5 frames doesn't amortise the CUDA launch/copy cost. CUDA decisively wins once the workload grows (see 4K below).

Backend comparison (BBB 4K, 200 frames)¶

Source: testdata/bbb/{ref,dis}_3840x2160_200f.yuv (BigBuckBunny 4K master, ffmpeg-encoded ref + libx264 CRF=35 round-trip distortion; see How to reproduce).

Notes:

• CUDA at 4K is the headline number — the RTX 4090 sustains 227 fps on 8-bit 3840×2160 with vmaf_v0.6.1.json, ~16× faster than the CPU + AVX-512 baseline.
• SYCL on Arc A380 is fp64-emulated (the A380 is a Gen12.7 part without native fp64). The 2.3× headline understates the SIMD path's potential on a fp64-native dGPU; revisit when an Arc B-series or Battlemage host lands. See backlog T7-17.
• Vulkan rows are historical; the backend was removed in ADR-0726. The performance bottleneck (dispatch-overhead-bound on NVIDIA, 14 fps matching CPU) contributed to the removal rationale.

CPU SIMD-ISA breakdown (576×324)¶

Selected via --cpumask (bits set = ISAs to disable).

AVX-512 over AVX2 buys another 2.24× on top of the AVX2 baseline on the 9950X3D (Zen 5 has 512-bit SIMD pipes). Pools match across all three configurations to within assertAlmostEqual(places=6) per the Netflix golden gate.

--precision overhead (576×324 CPU, 48 frames)¶

String formatting is not on the hot path; switching from %.6f (default per ADR-0119) to %.17g (--precision=max) changes only the JSON-emit stage.

Wall-time delta is in the noise (<1 % across all three), confirming that the per-frame cost of %.17g is negligible — the cost shows up in JSON byte-count, not in wall time. Use --precision=max whenever cross-backend numerical diffing or IEEE-754 round-trip determinism matters.