pytest suite audit — ai/, tools/vmaf-tune/, mcp-server/vmaf-mcp/ (2026-05-27)¶

Summary¶

Full pytest sweep of the three Python test suites that live outside python/test/ (the Netflix golden-data gate). Run against origin/master detached HEAD in an isolated worktree. No test files were modified.

Note: the 8 mcp-server collection errors only occur when mcp-server/vmaf-mcp/src is absent from PYTHONPATH; they resolve immediately once the correct path is added. The table above reflects results with the correct PYTHONPATH=ai/src:tools/vmaf-tune/src:mcp-server/vmaf-mcp/src.

Environment¶

Python  : 3.14.5
pytest  : 9.0.3
pytest-timeout : 2.4.0
PYTHONPATH : ai/src:tools/vmaf-tune/src:mcp-server/vmaf-mcp/src
Branch  : origin/master (detached, worktree agent-a2e3712a5695edabc)
Date    : 2026-05-27

Failures¶

1. tools/vmaf-tune/tests/test_adr_0543_backend_enforcement.py::test_adr_0543_per_feature_pinned_to_inactive_backend_fails¶

File: tools/vmaf-tune/tests/test_adr_0543_backend_enforcement.py:235

Error (truncated):

AssertionError: ADR-0543 regression: --feature motion_hip on --backend cpu exited 255 (expected 100)
stderr="/usr/local/bin/vmaf: unrecognized option '--backend'\nproblem loading feature extractor: motion_hip\n"
assert 255 == 100

Root cause: The installed vmaf binary at /usr/local/bin/vmaf does not recognise --backend. The test calls the system binary directly and expects exit code 100 (ADR-0543 structured per-feature conflict exit) but receives 255 (unrecognised option from an older build that predates --backend). This is an environment mismatch, not a code regression — the test is correct; the system binary is stale.

Classification: Environment / stale system binary; not a code bug.

Collection error (without mcp-server/vmaf-mcp/src in PYTHONPATH)¶

All 8 mcp-server test files fail to collect with:

ModuleNotFoundError: No module named 'vmaf_mcp'

because pytest is invoked from the repo root and mcp-server/vmaf-mcp/src/ is not on sys.path by default. Adding :mcp-server/vmaf-mcp/src resolves all 8 immediately.

Recommendation: Add a conftest.py at the repo root or a pytest.ini / pyproject.toml pythonpath entry that covers mcp-server/vmaf-mcp/src so pytest mcp-server/ works without a manual PYTHONPATH prefix.

Skipped tests¶

ai/tests/ — 1 skip¶

tools/vmaf-tune/tests/ — 15 skips¶

mcp-server/vmaf-mcp/tests/ — 2 skips¶

xfail tests¶

Both are correctly xfailed on non-macOS hosts.

Top 20 slowest tests (combined, across all suites)¶

No test outside test_bbb_e2e_v5_bug_cluster.py exceeded the 5 s mark. The vmaf-tune BBB end-to-end test at rank 1 is an integration test that runs a full ladder encode; it is expected to be slow.

Deprecation warnings (non-fatal, for awareness)¶

All 58 warnings in ai/tests/ are DeprecationWarning / UserWarning from upstream PyTorch / PyTorch Lightning:

1. torch.ao.quantization deprecated (removal in PyTorch 2.10) — ai/train/qat.py uses prepare_qat_fx / eager-mode quantization. Migration target: torchao pt2e API.
2. TorchScript ONNX exporter deprecated (removal in PyTorch 2.9 default) — multiple torch.onnx.export(...) calls across ai/ and tools/vmaf-tune/src/vmaftune/. Migration target: torch.export-based exporter + dynamic_shapes argument.
3. dynamic_axes deprecated when dynamo=True — companion to (2).
4. self.log() before Trainer attached — cosmetic PL warning, no impact.

Recommendations for follow-up fix PRs¶

P0 — Immediate (blocking CI confidence)¶

P1 — Short-term (hygiene)¶

P2 — Nice-to-have¶

Methodology notes¶

• pytest --timeout=60 --tb=short -p no:cacheprovider --durations=20
• All three suites run in parallel background processes against the same detached origin/master worktree.
• Netflix golden-data assertions in python/test/ were not touched and are outside scope.
• No test file was modified.