MobileSal saliency (legacy placeholder checkpoint)¶

vmaf_tiny_mobilesal_placeholder_v0 — the historical smoke checkpoint for the no-reference saliency feature extractor (mobilesal). The extractor runs a tiny ONNX saliency model over the distorted frame and emits the mean of its per-pixel saliency map as a scalar feature named saliency_mean. It is the scoring-side surface for Wave 1 §2.3 of the tiny-AI roadmap, the first half of backlog item T6-2 (T6-2a). Encoder-side ROI tooling (tools/vmaf-roi, per-CTU QP-offset sidecars) is shipped as T6-2b.

For production saliency use saliency_student_v2, the fork-trained default since 2026-05-15 (ADR-0444), or saliency_student_v1 (ADR-0286). Both keep the same input / saliency_map tensor names and run through the same feature_mobilesal.c extractor.

Upstream MobileSal weights remain deferred by ADR-0257 (CC BY-NC-SA 4.0, Google-Drive-walled, RGB-D); the fork-trained students provide the license-clean production path.

Upstream paper: Wu, Liu, Cheng, Lu, Cheng, "MobileSal: Extremely Efficient RGB-D Salient Object Detection", IEEE TPAMI 2021.

What the output means¶

The extractor emits a single feature named saliency_mean, one value per frame (the distorted frame is the input — saliency is no-reference).

Saliency-mean is not a quality score on its own — it is a content descriptor. Downstream consumers correlate saliency_mean against existing metric features (e.g. vmaf, lpips, psnr) to study how foreground-vs-background distortion affects subjective quality.

The full saliency map is computed internally to derive the mean; it is intentionally not exposed as a per-pixel feature in T6-2a. The encoder side (tools/vmaf-roi per-CTU QP-offset sidecar) consumes the same model in T6-2b and exports the map in encoder-native format.

Shipped checkpoint¶

The placeholder ONNX is deterministic (no doc_string, fixed producer_version, deterministic protobuf serialisation) so the sha256 stays stable across re-runs of the export script.

For content-dependent saliency, point the extractor at the production default model/tiny/saliency_student_v2.onnx (or model/tiny/saliency_student_v1.onnx). The placeholder is retained to keep the historical ABI / I/O-contract smoke path available.

Input / output contract¶

The C extractor binds tensors by name, so any future drop-in (real upstream MobileSal export, distilled student, etc.) must declare the exact same names:

inputs:
  input         float32[1, 3, H, W]   ImageNet-normalised RGB, NCHW
outputs:
  saliency_map  float32[1, 1, H, W]   per-pixel saliency in [0, 1]

H and W are dynamic — both the placeholder and the upstream graph match whatever resolution the C side feeds. ImageNet normalisation (mean [0.485, 0.456, 0.406], std [0.229, 0.224, 0.225]) is applied in the C side via the shared vmaf_tensor_from_rgb_imagenet() helper, identical to LPIPS's wiring (see lpips_sq_v1.md).

Usage — CLI¶

vmaf \
    --reference ref.yuv \
    --distorted dist.yuv \
    --width 1920 --height 1080 --pixel_format 420 --bitdepth 8 \
    --feature mobilesal=model_path=model/tiny/saliency_student_v2.onnx \
    --output score.json

The --feature argument takes name=option=value (colon-separated for more options); there is no --feature_params option. The output JSON gains a per-frame saliency_mean column alongside any other features requested in the same run. Combine with lpips and vmaf for the full saliency-quality picture:

vmaf --reference ref.yuv --distorted dist.yuv \
    --width 1920 --height 1080 --pixel_format 420 --bitdepth 8 \
    --feature vmaf \
    --feature lpips=model_path=model/tiny/lpips_sq.onnx \
    --feature mobilesal=model_path=model/tiny/saliency_student_v2.onnx \
    --output combined.json

Equivalently, set the model path via env var:

VMAF_MOBILESAL_MODEL_PATH=model/tiny/saliency_student_v2.onnx \
    vmaf --reference ref.yuv --distorted dist.yuv \
        --width 1920 --height 1080 --pixel_format 420 --bitdepth 8 \
        --feature mobilesal --output score.json

Usage — C API¶

#include <libvmaf/libvmaf.h>

VmafFeatureDictionary *opts = NULL;
vmaf_feature_dictionary_set(&opts, "model_path", "model/tiny/saliency_student_v2.onnx");
int err = vmaf_use_feature(ctx, "mobilesal", opts);
/* ... vmaf_score_pooled(ctx, ..., "saliency_mean", ...) for the per-frame mean */

Equivalent to setting VMAF_MOBILESAL_MODEL_PATH before vmaf_use_feature(ctx, "mobilesal", NULL).

Known limitations¶

With --bitdepth 10 or 12 together with --feature mobilesal the run aborts before scoring. The saliency model requires 8-bit ImageNet-normalised RGB; wider depths would require retraining and are not planned (ADR-0613 §P1-3). The message reads:

mobilesal: bpc=10 is not supported (8-bit only). The mobilesal extractor
requires 8-bit YUV input because the saliency model was trained on 8-bit
ImageNet-RGB. Use --bitdepth 8 or omit --feature mobilesal for HDR /
10-bit / 12-bit content.

Training and evaluation¶

The placeholder is not trained: it is a generated Conv + Sigmoid with fixed weights, so no training data, quality evaluation or correlation figure exists for it. Trained, evaluated saliency weights are on the saliency_student_v1 and saliency_student_v2 cards.

How the placeholder is regenerated¶

python ai/scripts/gen_mobilesal_placeholder_onnx.py          # rewrite model/tiny/mobilesal.onnx
python ai/scripts/gen_mobilesal_placeholder_onnx.py --check   # exit 1 if the file differs
# wrote model/tiny/mobilesal.onnx (330 bytes, sha256 f12263...)

The generator writes the ONNX file only; the sidecar JSON and the registry entry are committed files. The output is byte-identical to the shipped file, and --check fails when it is not. The sha256 in registry.json is verified before CreateSession.

Related¶

• lpips_sq_v1.md — sister full-reference DNN extractor; shares the YUV → ImageNet-RGB plumbing.
• ../roadmap.md §2.3 — Wave 1 MobileSal scope.
• saliency_student_v2.md — production default saliency weights for this extractor (ADR-0444).
• saliency_student_v1.md — initial fork-trained saliency student baseline (ADR-0286).
• ADR-0218 — design notes (smoke-only placeholder, scoring-vs-encoder split, scalar-vs-map output).
• ADR-0257 — first blocker (T6-2a-followup real-weights swap deferred): upstream MobileSal license, distribution and RGB-D mismatch.
• Research-0053 — upstream survey, licence analysis, and alternatives walk.
• ADR-0265 — second blocker: U-2-Net u2netp distribution + op-allowlist mismatch.
• Research-0055 — companion survey for ADR-0265.
• ADR-0286 — fork-trained production saliency-student path.
• ADR-0042 — tiny-AI doc-substance rule this page satisfies.