vmaf-tune predict, predictor training and the local sidecar¶

vmaf-tune predict estimates the VMAF of each shot of a video without encoding every shot to the end. It probe-encodes each shot, runs a learned ONNX predictor (or an analytical fallback), checks the result against real libvmaf scores on a few shots, and prints a verdict. The same predictors feed the fast, per-shot, ladder and auto paths. Overview: vmaf-tune.md.

Predict and validate¶

Run the predictor on a container source and check it on 8 shots:

vmaf-tune predict \
    --source source.mp4 \
    --codec libx264 \
    --target-vmaf 96 \
    --report-out predict-report.json

The command needs ffmpeg, ffprobe and the vmaf-perShot binary (--per-shot-bin) for shot detection. When vmaf-perShot is unavailable, pass --total-frames for the single-shot fallback.

The report is JSON, written to --report-out or to stdout:

Exit codes: 0 for gospel or recalibrate, 2 for fall_back, 1 when the source cannot be probed or no shots are detected. A predictor model that is a synthetic stub prints a warning on stderr; it is not authoritative for production CRF picks.

Flags¶

Without a calibration sidecar, --with-uncertainty degrades to low == high == point and the report is flagged uncalibrated. Background on the intervals: conformal VQA; predictor API: predictor.

Saliency features¶

--use-saliency decodes each sampled shot to temporary yuv420p, runs the saliency ONNX and feeds only saliency_mean and saliency_var into the predictor:

vmaf-tune predict \
    --source source.mp4 \
    --codec libx264 \
    --target-vmaf 96 \
    --use-saliency \
    --saliency-model model/tiny/saliency_student_v1.onnx

Train predictors from a corpus¶

vmaf_train.predictor_train trains the per-codec ONNX predictors. It belongs to the ai/ training package (pip install -e ai), the only place besides the ensemble kit where torch is installed (ADR-1886); vmaf-tune itself only loads the trained ONNX models. --corpus accepts one JSONL file or a directory of JSONL shards; directories are scanned recursively in sorted order, so the trainer can read .corpus/corpus_run/ directly:

python -m vmaf_train.predictor_train \
    --corpus .corpus/corpus_run \
    --codec libx264 \
    --output-dir .workingdir/evidence/predictor-real

Corpus handling:

• Rows are filtered per codec after schema aliases are normalised. The trainer accepts current corpus rows (encoder, crf, vmaf_score, bitrate_kbps) and older hardware-sweep rows (codec, q or cq, vmaf, actual_kbps).
• A codec with no usable rows falls back to the documented synthetic-stub corpus, and its model card records corpus.kind: synthetic-stub-*. A codec with rows in any shard records corpus.kind: real-N=<rows>.
• Richer predictor inputs are preserved instead of zero-filled: probe_i_frame_avg_bytes, probe_p_frame_avg_bytes, probe_b_frame_avg_bytes, saliency_mean, saliency_var, frame_diff_mean, y_avg and y_var. Older rows stay valid: missing probe-byte columns use deterministic bitrate stand-ins, and missing saliency or signalstats columns stay 0.0.

Local sidecar bias correction¶

vmaf-tune sidecar trains a small correction on your own host from the residuals between predicted and observed VMAF. It never uploads captures and never changes the shipped predictor. State is stored under ${XDG_CACHE_HOME:-~/.cache}/vmaf-tune/sidecar/. Background and algorithm: local sidecar training.

The four actions share these flags:

1. Inspect the current state:

   vmaf-tune sidecar status --codec libx264 --json
   

2. Record one observed encode. features.json is either a flat ShotFeatures object or { "features": { ... } }. The required fields are probe_bitrate_kbps, probe_i_frame_avg_bytes, probe_p_frame_avg_bytes and probe_b_frame_avg_bytes. Add --no-persist to update in memory without saving:

   vmaf-tune sidecar record \
       --codec libx264 \
       --features-json features.json \
       --crf 28 \
       --observed-vmaf 94.2
   

3. Or train in batch from a JSONL file with one observed encode per line:

   {"features":{"probe_bitrate_kbps":3000,"probe_i_frame_avg_bytes":10000,"probe_p_frame_avg_bytes":2000,"probe_b_frame_avg_bytes":1000,"width":1920,"height":1080,"fps":24},"crf":28,"observed_vmaf":94.2}
   

   vmaf-tune sidecar batch-record --codec libx264 --captures-jsonl captures.jsonl
   

4. Predict with the correction. The output lists the bare predictor score (base_vmaf), the sidecar correction and the final clamped score (sidecar_vmaf):

   vmaf-tune sidecar predict --codec libx264 --features-json features.json --crf 28 --json
   

See also¶

• Overview
• Corpus
• Fast path