CI overview¶
Run the local gate before you push, then let CI repeat it. This page tells a contributor which checks exist, which of them block a merge, and where to find the detail. The authoritative trigger and gate behaviour lives in the workflow files under .github/workflows/; the topic pages linked below carry the long procedures.
Before you push¶
Run the local subset of CI. It catches the usual formatter, lint and fast-test failures in seconds, against a 10-minute CI round trip:
make verify-all # HISS/context/evidence, duplicate implementations, docs gate
make dedupe-check # fast standalone AST clone scan
make format-check # clang-format + black + ruff, no writes
make lint # configured native + Python, shell, Markdown, Go and docs checks
make test-fast
bash scripts/ci/twin-drift-check.sh # .c/.cpp twin drift + stale source refs (ADR-1135)
pre-commit run --all-files # if .pre-commit-config.yaml hooks are installed
The format-check and pre-commit pair catches roughly the same surface as the pre-commit job of lint-and-format.yml.
The duplicate scan is deliberately separate from standardsctl audit: the audit baseline does not include AST clones. make verify-all, pre-commit, pre-push and the required Standards job therefore invoke standardsctl dedupe scan . explicitly. A finding is a hard failure; there is no origin, generated-code or historical-debt exemption.
| Topic | Page |
|---|---|
| Configure a lint build, read receipts, cppcheck models | Local lint |
| Whole-tree clang-tidy baselines and lanes | Tidy ratchet, measuring the lanes |
docs/state.md gates and rebase conflicts | state.md gates |
| Praetor documentation gate, text register, pin moves | Praetor gate |
| Display names of every check | CI job display names |
Workflows¶
The files under .github/workflows/ and what each is for. Which checks block a merge is decided by the aggregator; for the mapping of shortened names and conventions see CI job display names, and for every build lane, whether it is required and which ADR owns it, see ADR-1259.
Pull-request checks¶
| File | Purpose |
|---|---|
required-aggregator.yml | Single required-check aggregator (ADR-0313). |
ci-tier.yml | Reusable workflow every pull-request workflow calls first: decides the tier (ADR-2169); no trigger of its own. |
ci-escalate.yml | Re-runs the suite at the full tier when ci: full or autorelease: cut is applied. |
lint-and-format.yml | Pre-commit, clang-tidy (changed files plus the whole-tree ratchet, ADR-1142), cppcheck, mypy, registry validate, twin-drift gate (ADR-1135). |
standards-gate.yml | Required HISS and context verification and the fail-closed duplicate-implementation scan. |
rule-enforcement.yml | ADR-0100, 0106, 0108 and 0165 process gates. |
tests-and-quality-gates.yml | Netflix golden, sanitizers, tiny-AI, MCP, coverage, assertion density, the test-suite registry and tooling suite (Tooling Tests) and the Python package suites (Python Package Tests (<suite>)); which job runs which test suite: test suites. |
security-scans.yml | Semgrep, CodeQL, Gitleaks, Dependency Review. |
libvmaf-build-matrix.yml | Cross-platform, cross-backend libvmaf build matrix: 17 lanes, six of them required. |
build.yml | One all-backend build per OS (Linux Intel LLVM, macOS Clang+Metal, Windows MSVC+CUDA (full)), alongside the matrix; required since ADR-1297. |
go-ci.yml | Required Go modernization, vet, security scan, runner smoke and tests (ADRs 1238 and 1338). |
ffmpeg-integration.yml | FFmpeg plus libvmaf build (Linux GCC, macOS Clang, SYCL). |
ffmpeg-patch-stack.yml | Replays the cumulative FFmpeg patch series (see FFmpeg patch automation). |
upstream-consumers.yml | Unpatched upstream FFmpeg and GStreamer against the installed compat libvmaf.so.3, scores compared with the CLI and the base commit (full tier; see upstream consumers). |
sycl-parity.yml | SYCL parity on the self-hosted Intel Arc A380 runner (ADR-1177; see the runbook). |
docs.yml | Docs build. |
doxygen-public-api.yml | Doxygen build of the public C API; required since ADR-1297. |
docker-image.yml | Docker image build. |
release-dry-run.yml | Builds the release images and the vmaf-mcp distribution and SBOM without publishing (ADR-1595); required context Release Dry Run on pull requests (ADR-1687). |
windows-tester-bundle.yml | The Windows tester zips: the x64 zip on a pull request, all four on a push, published on dispatch; required context Windows Tester Zip (ADR-1687). |
dev-container-build.yml | PR-time build gate for dev/Containerfile (ADR-0819). |
helm-chart.yml | helm lint of the chart. |
rust-ci.yml | Rust crates: cargo fmt --all and clippy --workspace, cargo test --workspace, the golden smoke example and cargo-deny; the planner may skip the work, the gates vmafx-sys CI and cargo-deny are required. |
sanitizers.yml | Combined ASan and UBSan on full-tier pull requests and master pushes, TSan on master pushes, nightly fuzzing. |
praetor-docs.yml | Praetor's Documentation Governance gate for the docs:seo-portal facet; praetor-managed, not required. See Praetor gate. |
praetor-api.yml | Praetor's Go API Compatibility gate (go-apidiff over every Go module; no path filter). Praetor-managed; required through the aggregator (ADR-1506), its marker sits in standards-gate.yml. |
reuse.yml | Praetor's REUSE lint gate (reuse lint over the whole tree; no path filter), the one CI run of reuse lint. Written by praetor, its actions pinned to commits here; required through the aggregator (ADR-2784), its marker sits in standards-gate.yml. See Praetor gate. |
scorecard-policy.yml | OpenSSF Scorecard PR policy (ADR-1247). |
pr-type-label.yml | Derives a type:* label from the Conventional-Commit prefix of the PR. |
Scheduled, release and watcher workflows¶
| File | Purpose |
|---|---|
nightly.yml | Nightly jobs, including the whole-tree clang-tidy ratchet of the cpu lane. |
nightly-bisect.yml | Nightly bisect-model-quality smoke against a committed fixture timeline. |
fuzz.yml | Nightly libFuzzer smoke over every harness under core/test/fuzz/. |
scorecard.yml | Weekly OpenSSF Scorecard scan. |
e2e-k8s.yml | Kubernetes integration harness for the VMAFx runtime; label- and schedule-gated. |
release-please.yml | On each push to master: opens or updates the release PR, or creates the draft release (ADR-1127, ADR-1128). |
supply-chain.yml | Build provenance, Sigstore signing and SBOM when a release draft is published. |
dev-container-publish.yml | Builds, pushes and signs the canonical dev container image to GHCR, only into a private package (ADR-1564). |
docker-publish-production.yml | Builds, pushes, signs and SBOMs the production image on release publication. |
docker-publish-operator-node.yml | The same for the VMAFX Go service images. |
published-rc-licence-companions.yml | Manual: notices, <tag>-source companions and SBOMs for the images and release files published for 1.0.0-rc.1 and rc.2 (ADR-1578). |
docker-publish-tester.yml | Builds, tests, signs and attests the tester image; its pull-request run (the amd64 image) is the required context Tester Image (ADR-1687). |
macos-tester-bundle.yml | Builds, tests, attests and publishes the macOS arm64 tester bundle. |
upstream-watcher.yml | Polls FFmpeg master for upstream-blocked features (upstream watchers). |
upstream-ffmpeg-hip-hwdec-watcher.yml | Weekly watch for an FFmpeg ROCm/HIP hwdec context type (ADR-0448). |
upstream-netflix-645-hdr-model-watcher.yml | Watches Netflix/vmaf#645 and the upstream HDR model (ADR-0448). |
upstream-netflix-955-watcher.yml | Watches Netflix/vmaf#1494, the upstream fix for #955 (ADR-0448). |
research-radar.yml | Weekly digest of changes in the public video-quality sources of the research radar (ADR-2171). |
Python type-check gate¶
Python Lint is a required, fail-closed check. It installs the reviewed requirements/locks/mypy.txt lock and runs the same merge-base gate as the local mypy-local pre-push hook:
The gate checks added, copied, modified, renamed and type-changed tracked *.py paths under ai/ and scripts/. It reports only findings absent from the selected merge base, but an analysis crash, a missing tool, base or file, or an unattributable nonzero status fails the job. ai/src/ is checked separately with --explicit-package-bases so each module has one canonical identity.
Pull requests compare with origin/master. A master push compares with the event's previous commit, so hosted post-merge validation covers the pushed range. See ADR-1310 and the detailed hook contract.
Draft pull requests defer heavy CI¶
A draft PR cannot satisfy the required aggregator. Per ADR-0331, every pull_request-triggered workflow is gated to skip while the PR is a draft, except the aggregator, which explicitly fails drafts:
- each workflow's
pull_request:block liststypes: [opened, synchronize, reopened, ready_for_review]; - the first job of the workflow,
tier, carries anif:clause of the formgithub.event_name != 'pull_request' || github.event.pull_request.draft == falseand every other job needs it (ADR-2169; before that each job carried the clause itself). The workflows that cannot carry it are the two praetor-locked ones, listed under Which jobs run when.
What this means for contributors:
- A draft PR cannot satisfy the required aggregator. The aggregator starts and fails with a request to mark the PR ready. Heavy jobs remain skipped until ready-for-review, so skipped draft-era checks are never mistaken for completed validation.
- Promoting the draft to ready-for-review fires CI exactly once. GitHub's
ready_for_reviewevent re-triggers the workflows; latersynchronizeevents on the now-ready PR fire CI as before. - Pushing to
masteris unaffected. The job-levelif:clause short-circuits totruewhen there is no PR object (for example onpush:events).
To preview CI status before merging, mark the PR ready-for-review. You can flip back to draft afterwards; the next ready_for_review fires a fresh matrix.
Which jobs run when (ADR-2169)¶
A pull request does not owe the whole suite. The tier of an event decides which required contexts run; one file, .github/ci-tier.json, defines the tiers, and the workflows and the aggregator both read it.
| Event | Tier | What runs |
|---|---|---|
| Pull request from a fork | full | everything |
| Pull request from this repository (Renovate included) | light | lint, format, the fast suite and the governance gates |
Own pull request with the label ci: full | full | everything |
Release pull request (release-please-- branch, bot author or the maintainer account with a release-only diff) | release-light | Release Script Contract |
Release pull request with the label autorelease: cut | full | everything |
| Draft pull request | none | nothing, but the aggregator, which fails it |
Push to master, dispatch, schedule | full | everything |
| Push to another branch, push of a tag | none | nothing |
The light tier is every required context except the full_only list of ci-tier.json: the platform legs of the build matrix (the Ubuntu gcc+DNN and clang+DNN legs stay), the all-backend Build lanes, Windows and macOS legs, GPU builds, coverage, sanitizers, the dev container, docker image, FFmpeg integration, tester and release dry-run lanes, and the self-hosted hardware lanes. Those run on the master push, so a break in them is found there, after the merge train landed the change. To see one on a pull request, add the label ci: full.
What a contributor sees:
-
Every workflow shows a job
CI tier (<workflow>) / Decide the CI tierfirst; the other jobs of the workflow appear when it completes (GitHub creates a job that needs another only then), and the aggregator waits for the tier decisions before it judges which contexts are missing. -
A job the tier does not run shows as skipped. The aggregator accepts a skipped or absent context the tier does not owe, and still fails one that ran and failed. The matrix legs of
libvmaf-build-matrix.ymlare the one place a job starts and stops: a matrix cannot be filtered per leg, so a full-tier leg on a light-tier pull request ends after a shallow checkout with a notice. - Adding the label
ci: full(orautorelease: cutto the release pull request) startsci-escalate.yml, which re-runs the latest run of each workflow on the head commit. Every re-run decides its tier again from the live labels, so the skipped jobs now run and the aggregator waits for them. Removing the label does not cancel anything; the next push is light again. - The release pull request is refreshed on every merge to master. Without the cut label it runs
Release Script Contractonly, which is the check that proves the cut ran. The maintainer appliesautorelease: cutwhen the release is to be cut, and the full suite runs once on the exact head. - A pull request from a fork always runs the full tier: the head repository is compared with
github.repositoryinscripts/ci/ci_tier.py. - A branch named
release-please--...is not enough to be the release pull request:scripts/ci/release-pr-exempt.shchecks the author and, for the maintainer account, that the diff touches only release files (ADR-1151, ADR-1388). Anything else is an ordinary own pull request.
How a workflow takes part: its first job is tier, a call of ci-tier.yml. Light-tier jobs gate on needs.tier.outputs.light == 'true', full-tier jobs on needs.tier.outputs.full == 'true'; a planner workflow gates its impact job and its gate job, and a gate job is always() && needs.tier.outputs.<tier> == 'true'. The tier job also carries the draft gate of the previous section. The exceptions (jobs without the tier, each with a reason and an expiry) are the untiered_jobs of ci-tier.json: the aggregator, Release Script Contract, the label and escalation workflows, the opt-in e2e gate, and the praetor-managed praetor-api.yml and praetor-docs.yml. The last two are locked byte for byte by praetorctl audit. Since the pin 7458a220e1c9 (praetor#815) they push only on master and listen for opened, synchronize, reopened and ready_for_review, so they no longer start on a push to another branch. On a draft their first step fails closed and every later step is skipped: the job starts and reports a red check but does no work. A job-level gate cannot be added to a byte-locked file, so they stay in untiered_jobs, and test_praetor_managed_jobs_stop_on_a_draft_before_any_work holds the step shape. push_branch_exceptions is empty.
Adding a required context: add it to the aggregator required list and the # required-aggregator marker as before, and to full_only or always in ci-tier.json if it is not a light-tier context. The routing contract fails when the workflows and the file disagree.
The routing contract¶
scripts/ci/tests/test_ci_routing_contract.py is the proof. It builds synthetic events (own, fork, Renovate, release with and without the cut label, a person on a release branch name, draft, master push, a push to a feature branch, a tag, an unrelated label, an escalation label), works out with scripts/ci/ci_router.py which jobs of the real workflow files run (scripts/ci/ci_expressions.py evaluates the if: expressions as the Actions documentation defines them, and the tier decision is the real ci_tier.py), and compares that with ci-tier.json. Six more cases plant a defect in a copy of the workflows and require the contract to fail. To see what the contract checks against another tree, point it at a workflow directory:
CI_ROUTING_WORKFLOWS_DIR=<tree>/.github/workflows \
python3 -m unittest scripts.ci.tests.test_ci_routing_contract
The aggregator side is test_ci_aggregator_tier.py, the decision is test_ci_tier.py, and the escalation is test_ci_escalate.py, all in scripts/ci/tests/.
Renovate¶
renovate.json runs in a weekly window (Monday before 6am, Europe/Vienna). A first catch-all rule puts every minor, patch, digest and pin update that no other rule groups into one pull request; the existing ecosystem rules keep their own groups, automerge and review settings; major updates stay individual and manual; rebaseWhen is conflicted, so a Renovate branch is not rebuilt each time master moves. Security updates (vulnerabilityAlerts) are opened at any time and are not grouped. Renovate pull requests come from this repository and run the light tier.
Gates judge their own matrix leg¶
A matrix job's needs.<job>.result is the aggregate of every leg, so a gate that reads it fails for a leg it does not name. The gates that share a matrix (Linux Intel LLVM, macOS Clang+Metal and Windows MSVC+CUDA (full) in build.yml; FFmpeg Ubuntu gcc and FFmpeg macOS clang in ffmpeg-integration.yml) therefore run scripts/ci/gate_leg_result.py with the name of their own <check name> work job. It reads the run's jobs (gh api repos/<repo>/actions/runs/<id>/jobs --paginate, which needs actions: read) and passes when the planner succeeded and either the leg was selected and concluded success, or it was not selected and is absent or skipped. A missing, unfinished or ambiguous own job fails. A new gate on a shared matrix does the same; scripts/ci/tests/test_gate_leg_result.py fails on two gates that read one matrix job's aggregate.
CI impact routing (ADR-1140)¶
Required checks do not decide whether they apply from a workflow-level paths: or paths-ignore: filter. Every workflow that hosts a check named in required-aggregator.yml starts on every non-draft PR and every push to master; the first step of each required job runs the planner:
python3 scripts/ci/plan-ci-impact.py --event pull_request \
--base <base-sha> --head <head-sha> --github-output "$GITHUB_OUTPUT"
The planner diffs the event's exact revisions (the merge-base of head and base for a PR, the exact before..head for a push) and maps the changed paths onto the selectors declared in .github/ci-impact.json. There are 19 selectors:
| Selector | Owns | Gates |
|---|---|---|
c_core | core/, ffmpeg-patches/, model/, testdata/, golden fixtures | Build legs, sanitizers, cppcheck, CodeQL C/C++, assertion density, Tidy Ratchet |
python | python/, compat/, mcp-server/, tools/, dev-llm/, scripts/**/*.py, requirements/ | Inherited by the composite selectors below |
ai | ai/, model/ | Inherited by tiny_ai and python_lint |
go | cmd/, pkg/, internal/, api/, proto/, gen/, *.go, go.mod | Inherited by go_checks |
golden_harness | c_core plus python | Netflix golden tests, coverage gate |
tiny_ai | c_core plus ai plus python | Tiny AI (DNN suite and ai/ pytests) |
python_lint | python plus ai | CodeQL Python |
go_checks | go plus c_core | Go vet, security scan, native and ORT smoke, and Go tests |
docs | docs/, mkdocs.yml, *.md, changelog.d/ | Docs build |
actions | .github/ | CodeQL Actions, FFmpeg patch stack |
docker_image | c_core plus the Dockerfile and Python requirements | Docker image build |
dev_container | c_core, python, ai, go, shell plus dev/ | Dev container build |
doxygen | core/include/libvmaf/, the public-API Doxyfile | Doxygen public API |
helm | deploy/helm/ | Helm chart |
tester_image | docker/Dockerfile.tester, tools/rc1-tester/, the toolkit install scripts, build-config.env, the licence inputs; own paths only (ADR-1700) | Tester Image (ADR-1687) |
windows_tester_zip | The Windows zip build scripts, its lock file and the Windows inputs under tools/rc1-tester/image/; own paths only (ADR-1700) | Windows Tester Zip (ADR-1687) |
rust | bindings/, Cargo.*, core/src/feature/rust/ | Rust CI (path-filtered, not required) |
shell | *.sh | Not required, still path-filtered |
container | Dockerfile*, dev/, docker/, deploy/, .devcontainer/ | Not required, still path-filtered |
Steps gated on a selector that is not impacted are skipped and the job emits ::notice::<selector> not impacted (mode=... reason=...) before reporting success, so the aggregator always sees a real conclusion with a real reason.
The planner fails closed. Each of these produces mode=full, which sets every selector true (the behaviour before ADR-1140):
- an unknown top-level path;
- any status other than add or modify (delete, rename, copy, mode change);
- a change to a CI-authority file (the map, the planner,
scripts/ci/**, the workflows hosting required contexts,.pre-commit-config.yaml,Makefile,.clang-tidyand the like); - a missing merge-base, a non-linear push or an over-large diff.
One declared exception (ADR-1700): a selector with "own_paths_only": true is not set by the fallback itself. When the changed paths are known, it is true only if one of them matches its own patterns; when they are not (a dispatch, a schedule, a diff that could not be read), it stays true. Only tester_image and windows_tester_zip carry it, it needs patterns and no inherits, and test_ci_impact.py fails on any other use. A change under scripts/ci/ alone therefore runs every other gate and neither tester build.
Run it locally:
python3 scripts/ci/plan-ci-impact.py --event pull_request \
--base "$(git merge-base origin/master HEAD)" --head HEAD --print
python3 -m unittest scripts/ci/tests/test_ci_impact.py # map and tree contract
Adding a top-level directory or file? Add it to known_prefixes or known_files (and to a selector if a required check owns it); the contract test fails otherwise, because an unknown path would silently force full mode on every PR that touches it.
Master push runs¶
Master commits land every few minutes through a local merge train. A push to master must not cancel the runs of the previous master commit, or no master commit ever gets a complete verdict (ADR-1673).
- Every workflow triggered by a push to
masterputs the SHA in its group on master and keeps the ref elsewhere:
group: <name>-${{ github.ref == 'refs/heads/master' && github.sha || github.ref }}
cancel-in-progress: ${{ github.ref != 'refs/heads/master' }}
On a PR ref the group is still the ref and a newer push cancels the older run; on master each commit has its own group and nothing cancels it. - The merge train keeps one sentinel master commit's runs alive and cancels the other superseded master runs itself through the API. GitHub's per-ref cancellation is not involved on master. - The Required Checks Aggregator follows the same rule. Re-running the aggregator of an old master head no longer cancels the newest one. On a PR ref the old warning stands: re-running an older head's aggregator cancels the current head's. - Blocks that must serialise keep their group and are listed with a reason in scripts/ci/tests/test_master_concurrency_contract.py: dev-container-publish.yml (registry publish), release-please.yml (one release PR per push), scorecard.yml (attestation publish) and the deploy job of docs.yml (one Pages deployment). - A new workflow that triggers on a push to master takes the SHA form or is added to that list with a reason. Run the check with python3 -B -m unittest scripts/ci/tests/test_master_concurrency_contract.py.
Required-checks aggregator¶
The single required check on master branch protection is the Required Checks Aggregator (ADR-0313). The required array in required-aggregator.yml is the merge gate in full: a check missing from it can be red while the merge button stays green (ADR-1297).
It runs on every PR and master push. Draft PRs fail immediately; ready PRs poll for the named sibling check runs to reach a terminal state and accept success, skipped or neutral per check. Results predating the current run are excluded, so skipped draft-era checks cannot mask ready validation.
On a master push the aggregator reads only the push's own check runs: a check run whose check suite belongs to a pull-request workflow run on the same commit is left out. The merge train lands by fast-forward, so a landed commit is also the head of its pull request, and the train cancels that pull request's runs once it lands; without the filter those cancelled runs of the pull-request-only gates (Deliverables Checklist, docs/state.md Gate, Silent-Revert Guard and others) read as failures of every master push. A check run of a workflow run on another branch is left out too: release-please opens its release-notes branch at the master head, and the push runs there of the workflows without a branch filter are cancelled as superseded. Check runs of other apps, such as code scanning, always count. A pull-request aggregator reads every check run on its head as before. scripts/ci/tests/test_aggregator_event_scope.py runs the embedded script on both events.
Release legs¶
Three contexts gate what the release and tester workflows build on a pull request (ADR-1687). All three must report success; a missing or skipped one fails the aggregator.
| Context | Workflow | Does work when | Required on |
|---|---|---|---|
Tester Image | docker-publish-tester.yml | the planner selects tester_image | pull requests and master pushes |
Windows Tester Zip | windows-tester-bundle.yml | the planner selects windows_tester_zip | pull requests and master pushes |
Release Dry Run | release-dry-run.yml | always; scripts/ci/release-dry-run-plan.sh picks the groups | pull requests only (the workflow has no push trigger; the aggregator list pullRequestOnly drops it from other runs) |
These are full-tier contexts, with one declared exception (ADR-2198): Windows Tester Zip is an own-input lane (own_input_lanes of .github/ci-tier.json), so a pull request from this repository also runs it, but its planner builds only what the diff touches. Selector windows_tester_zip builds the x64 zip; selector windows_tester_zip_sycl (everything the x64 zip reads, plus sycl-rows.json, prepare_build.py, build-config.env and the SYCL scratch ratchet list) adds the x64 SYCL zip. The arm64 and CUDA zips, Tester Image's arm64 leg and the macOS bundle still build on the master push or a dispatch only, which is why a cut checks them (scripts/release/check-candidate-legs.py, see the release guide). For the other contexts a pull request from this repository runs them only with the label ci: full (see Which jobs run when), and the master push is where they run for every change. An unselected tester run passes in about a minute. Which inputs select them and how to reproduce a failure: verifying the release and tester workflows.
On a push to master the two tester gates need the whole push run of their chain, so when the push touches their inputs the master aggregator waits for both image architectures and all four Windows zips, and a red push leg turns it red. That is intended (ADR-1687, confirmed in ADR-1700): a broken tester package on master is visible in the same place as every other red check.
Go checks¶
The go vet + go test job is required under ADR-1238. Its native build, security scan, runner smoke and tests run for Go or core and model inputs; unrelated documentation changes report success after an explicit impact notice. A failing gosec scan blocks merging even though it prevents later Go tests from running. The scan is make lint-go, the target make lint runs, so run it before pushing a Go change; its flags and the reason for each sit next to the target in the Makefile. The job also starts on ready-for-review events, so draft-era results cannot replace the current validation run.
Before installing native build dependencies, the job runs go fix -diff ./... under the exact go.mod toolchain. Per ADR-1338, any available source rewrite is a blocking failure; CI never mutates the checkout. Run make go-fix, repeat if the Go tool reports cascading fixes, and finish with make go-fix-check locally.
Hardware-dependent lanes¶
Two checks depend on self-hosted hardware. Both fail closed when their lane is enabled and accept a skip only while it is disabled.
| Check | Switch | While disabled | While enabled | Runbook |
|---|---|---|---|---|
SYCL Parity (Arc A380) (ADR-1177) | repository variable SYCL_ARC_RUNNER_ENABLED | An absent or skipped job is accepted. | The job must report success; a skip, which is what the loud probe failure in sycl-parity.yml produces when the runner is unregistered, offline or the probe token is rejected, fails the aggregator. | ci-self-hosted-sycl.md |
Coverage GPU (ADR-1319) | GPU_COVERAGE_ENABLED plus the complete self-hosted,linux,gpu-full online label set | Absence or skip is permitted. | Coverage GPU must report success; probe failure, absence, skip or neutral blocks the aggregator. | self-hosted-runner.md |
Coverage GPU is a distinct gpu-full capability: a hosted job checks the variable and the label set before the hardware job can be dispatched. The Arc-only runner must not be relabelled to satisfy this CUDA plus SYCL contract.
Twin-drift gate¶
core/ carries same-directory .c/.cpp twin pairs left by the C++23 migration (ADR-0729). Twice a fix landed on one twin and never reached the other, and twice a rename (mem.c to mem.cpp, dict.c to dict.cpp) left a stale path in a build file that only nightly or opt-in lanes configure. ADR-1135 turns both into a blocking, required check: Twin Drift (job twin-drift-check) in lint-and-format.yml, backed by scripts/ci/twin-drift-check.sh.
What fails¶
The gate fails when either holds:
- A same-directory
.c/.cpppair exists and one side is compiled by no build file (meson.build,setup.py,*.pyx), unless that side is listed inscripts/ci/twin-drift-allowlist.txtwith a reason. - Any build file names a source path (
.c .cpp .cc .cxx .cu .hip .m .mm .metal .pyx) that does not exist in the tree.
How references are resolved¶
| Build-file form | Resolution |
|---|---|
'../src/x.c', 'x.c' | relative to the build file's directory |
src_dir + 'x.c' | through the src_dir = './.../' assignment in the same file |
_m + '_parity.c' (prefix is not a literal directory) | suffix search over git ls-files; reported as NOTE |
os.path.join("..", "core", "x.c") | joined; identifiers resolve through assignments |
output: 'gen.c', '@PLAINNAME@.c' | skipped; generated files |
/abs/path.c | skipped; toolchain-provided |
# ... comments | ignored (quote-aware) |
Clearing a failure¶
- Stale source reference: fix the path in the build file. There is no allowlist for this class. If the parser genuinely cannot model a construct, append
# twin-drift-ignore: <reason>to that line; the reason is mandatory and the line is reported asNOTE. - Dead twin side: wire the side into a build file, delete it, or add a row
path reasonto the allowlist. Rows without a reason, rows whose file is gone, whose side is compiled again, or whose pair no longer exists fail the gate, so the allowlist cannot rot.
Sides compiled only by test or fuzz build files are printed as INFO (non-failing): that is the drift-risk shape to keep an eye on when touching one of them.
The local run is identical to CI, takes about two seconds, needs no build, and is also wired as a pre-push hook:
bash scripts/ci/twin-drift-check.sh
bash scripts/ci/tests/test-twin-drift-check.sh # 24 fixture cases
Lint, standards and bookkeeping gates¶
These sections keep their anchors for existing links; the content lives on the pages named here.
Whole-tree lint ratchet (ADR-1142)¶
CI bounds clang-tidy findings for every file with a per-lane baseline (scripts/ci/tidy-baseline-<lane>.json): a file may never exceed its baseline, and a cleaner file must tighten it in the same PR. The required context is Tidy Ratchet (the cpu lane); the cuda, sycl, hip and arm64 lanes are measured in the dev container and are not PR-required. Rules, commands and exit codes are in Tidy ratchet; measuring is in measuring the lanes.
Carve-outs still open after ADR-1142¶
The open scope restrictions of the lint configuration (which paths Tidy Changed excludes, and what blocks each) are in the carve-out table of the tidy ratchet page.
Source ADR citations (ADR-1311, ADR-2200)¶
scripts/ci/check-source-adr-citations.py binds each plain ADR-NNNN in source and build files to the one file docs/adr/NNNN-*.md. It derives the binding from the tree on every run, so citing an ADR needs no registry edit; scripts/ci/source-adr-citations.json holds only the hand-written retired and fixtures records. The gate fails on a number that has no ADR file and no retirement record (file the ADR, or audit and record the retirement), on a retired number that has a file again, and on a fixture number cited outside its recorded paths.
Local lint build profile and receipts¶
make lint against a configured build, its receipt directory and the cppcheck model files are in Local lint.
Praetor documentation gate¶
make docs-lint and make docs-figures are praetor's Documentation Governance gate, and praetor owns several files byte for byte. See Praetor gate.
Moving the praetor pin¶
PRAETOR_REF in standards-gate.yml names the engine CI installs; the move procedure is in Praetor gate.
Resolving a docs/state.md rebase conflict¶
Run python3 scripts/dev/resolve-state-md-conflict.py docs/state.md mid-rebase; see state.md gates.
Bug-status hygiene gate (ADR-0165 / ADR-0334)¶
A fix: PR, a bug title or a close keyword must touch docs/state.md (rule 15 of the agent hard rules), or say no state delta: REASON in the body. See state.md gates.
Warnings are errors (ADR-2170)¶
HISS-10 asks for zero warnings. A leg that prints none turns warnings into errors, so the pull request that adds one fails that leg. The switch is one script, scripts/ci/werror-args.sh, called from the leg's meson setup line ($(scripts/ci/werror-args.sh "${{ matrix.werror }}") in the build matrix, werror: true on the matrix row). With true it prints -Dwerror=true, which is -Werror on every C and C++ compile, and the linker's own switch in -Dc_link_args / -Dcpp_link_args (-Wl,--fatal-warnings for GNU ld, lld and MinGW; -Wl,-fatal_warnings for Apple's ld64). With msvc (a leg that builds with cl.exe and link.exe) it prints -Dwerror=true alone: Meson turns it into /WX on every cl.exe compile and -WX on every link.exe link (Meson 1.12 adds the linker's fatal-warnings switch itself whenever werror is set), and core/src/meson.build adds --Werror all-warnings to every nvcc fatbin. The MSVC legs run their steps under cmd, so a shell: bash step named Warnings-as-errors arguments (id: werror) calls the script and the configure step appends ${{ steps.werror.outputs.args }}. lib.exe, which archives the static libraries of those legs, has no fatal-warnings switch in Meson; its warnings (none so far) show in the leg's log. Any other value prints nothing, except a typo, which exits 2. Rust has its own gate (cargo clippy -- -D warnings). Release and container image builds do not use the script: a compiler newer than the one a leg pins must not stop a release over a new diagnostic.
Praetor's build-warnings gate (HISS-10, praetorctl audit) reads the same requirement from the workflow files, but only as a literal on the build command: it cannot see what werror-args.sh prints, so it reads the gated legs as ungated too. Each workflow with a lane it reads that way is declared in .config/lint-exceptions.d/HISS-10.toml with a reason and an expiry (Praetor gate); spelling the switch where the gate reads it and gating the legs below remove those entries.
A fix for a warning changes no computed value and suppresses nothing: no -Wno-*, no #pragma ... ignored, no flag removed to hide a class. A diagnostic that is a defect of the tool needs a declared exception (one file, one rule, a reason, an expiry).
Legs that are gated¶
| Workflow | Legs | Toolchain family |
|---|---|---|
libvmaf-build-matrix.yml | Ubuntu gcc, gcc static, gcc+DNN, CUDA, CUDA static, HIP | gcc 14, nvcc |
libvmaf-build-matrix.yml | Ubuntu clang, clang+DNN, ARM clang, macOS clang, macOS clang+DNN, macOS Metal | clang 22, Apple clang |
libvmaf-build-matrix.yml | Ubuntu SYCL, SYCL+CUDA | icx / icpx |
libvmaf-build-matrix.yml | Windows UCRT64 | MinGW gcc |
libvmaf-build-matrix.yml | Windows MSVC+CUDA, Windows ARM64 MSVC | MSVC cl.exe / link.exe (x64 and ARM64), nvcc |
build.yml | Windows MSVC+CUDA (full) | MSVC cl.exe / link.exe, nvcc |
sanitizers.yml | ASan+UBSan, TSan | clang 22, lld |
go-ci.yml, rust-ci.yml | the libvmaf build the Go and Rust jobs link | gcc |
ffmpeg-integration.yml | the libvmaf build of the Ubuntu gcc, macOS clang and SYCL legs | gcc, clang, icpx |
Legs that are not gated yet¶
| Leg | Warnings (master 70d6dd0a5 unless stated) | What is left |
|---|---|---|
| Windows MSVC+SYCL (icx-cl) | 4,361 on the gate commit (2026-10-07, job 112842753545) | -Woverriding-option of the strict FP line (2,268: /fp:precise /Qfma-, and the SYCL -fp-model=precise -ffp-contract=off of the Windows icpx), /experimental:c11atomics unused by icx-cl (1,957), the C runtime's deprecated calls (131 at 52 sites: getenv, fopen, tmpfile, strdup), 3 unused functions, 2 ignored -ffp-contract=off |
| Dev Container Build | 204 | third-party sources built in the image (vpl-gpu-rt -Wstringop-overflow, FFmpeg) and gcc's LTO "serial compilation" note |
| Docker Image Build, Tidy Ratchet, Cppcheck, CodeQL, Coverage Gate and the other jobs that compile libvmaf for analysis | 1 to 12 | the same sites as the gated legs; they gate once the train has landed and a master run shows 0 |
What has been proven¶
Each toolchain family failed a planted warning once and passed again once it was removed (2026-10-07, the same build directories configured with the gate's arguments): gcc, clang, the MinGW cross compiler, clang for aarch64 and icx / icpx for -Wunused-variable and -Wunused-function in a C and a C++ translation unit; nvcc (--Werror all-warnings, diagnostic #177-D) and hipcc (-Werror, -Wreturn-type) in a device kernel; GNU ld (--fatal-warnings, "requires executable stack"). Not proven on this host, only on the CI run of the pull request that gates them: Apple clang and ld64 (-fatal_warnings), and lld (--fatal-warnings, the sanitizer legs: the compiler half is proven by clang, no default-on lld warning was found to plant).
The MSVC legs were proven on CI, on throwaway branches of the gate commit dispatched with workflow_dispatch (2026-10-07). float planted = 0.1; in core/test/test_picture.c failed Windows MSVC+CUDA, Windows ARM64 MSVC and Windows MSVC+CUDA (full) with C4305 as error C2220 (runs 37635562015 and 37635566701); #pragma comment(linker, "/VMAFXPLANTEDDIRECTIVE") failed the test_picture.exe link of Windows MSVC+CUDA and Windows ARM64 MSVC with LNK4229 as error LNK1218 (run 37635571313). The gate commit itself passed all three legs with werror : true in the Meson summary, -WX on the link lines and no warning in the logs (run 37635551653: the Windows MSVC+CUDA build, and the Windows ARM64 MSVC build with its 355 fast tests; run 37635557102: the Windows MSVC+CUDA (full) build and its CPU tests).
The macOS Metal leg links every target with the Objective-C++ compiler, and Meson 1.12 then names -lc++ twice (224 ld64 warnings per run). The clang++ driver adds the library itself, so duplicates are harmless; core/src/metal/meson.build passes -Wl,-no_warn_duplicate_libraries to those links, with the reason beside it, and the leg is gated. The deliberate register spill of the SYCL self-test kernel is compiled through core/src/sycl/run_captured.py, which prints the device compiler's output only when the compile fails.
Adding a leg¶
- Build the leg's configuration on a clean tree and count the warnings by unique (file, line, flag), not by lines of log (a header diagnostic repeats once per translation unit that includes it).
- Fix every site; run the Netflix golden gate and the fast suite.
- Put
werror: trueon its matrix row (or add$(scripts/ci/werror-args.sh true)to itsmeson setup;werror: msvcand theid: werrorbash step for an MSVC leg), move its row from the second table to the first, and runpython3 -m unittest scripts/ci/tests/test_werror_args.py. The test fails a build-matrix row or an MSVC leg that is neither gated nor listed above. - Prove the gate once: plant
int planted(void) { int unused; return 0; }(for MSVC,float planted = 0.1;, C4305) in a throwaway branch and see the leg fail; record the run in the pull request.
Flaky legs (2026-09-04 audit)¶
An audit of CI runs on master across all workflows found two leg reliability issues, addressed under epic #1236. Counts were taken with gh run list --branch master --limit 40 on 2026-09-05:
| Leg | Symptom | Frequency | Cause | Status |
|---|---|---|---|---|
tests-and-quality-gates.yml / MCP Smoke (Embedded C + Python Server) | test_mcp_smoke TIMEOUT 60.04s, killed after test 15 (test_uds_roundtrip) | 1 of 40 master runs (27 passed, 11 cancelled, 1 failed) | stop_uds() closed the AF_UNIX listener without shutdown(); on Linux that never wakes a worker already blocked in accept(2), so pthread_join() hung. A race: only when the worker re-entered accept() before close(). | Real bug, fixed in ci/flaky-legs-1236 (T-CI-MCP-SMOKE-TIMEOUT-2026-09-04) |
build.yml / libvmaf-build-matrix.yml macOS brew install ... llvm | Homebrew bottle download failure | 0 of 40 build.yml runs (25 passed, 14 cancelled on master; 0 download failures) | GitHub-hosted runner network or Homebrew CDN | Infrastructure; hardened with a 3-attempt retry plus brew fetch --retry (T-CI-MACOS-BREW-LLVM-FLAKE-2026-09-04) |
release-please.yml | Fails on every master push | 3 of 3 master runs in the window | Missing GitHub App credentials | Infrastructure; tracked in #1289 |
| Every other workflow on master | None | 0 failures in the last 40 master runs (one CI run was cancelled by a superseding push, not failed) | n/a | Nothing to fix |
MCP smoke timeout (run 33916590280, commit 0a8727ca7)¶
The MCP Smoke (Embedded C + Python Server) job hit the 60-second Meson timeout during test_uds_roundtrip.
- Root cause: in
core/src/mcp/mcp.c,stop_uds()calledclose(server->uds_listen_fd)withoutshutdown(server->uds_listen_fd, SHUT_RDWR)first. On Linux, closing a listeningAF_UNIXstream socket does not unblock a concurrentaccept(2)in the worker thread (vmaf_mcp_uds_thread_main), leaving the thread asleep in the socket wait queue whilestop_uds()waited indefinitely onpthread_join(). - Measured passing distribution: across 20 iterations
test_mcp_smoketakes about 0.03 s (range 0.01 s to 0.05 s). The 60 s timeout was a true deadlock, not a slow test. - Fix:
shutdown(server->uds_listen_fd, SHUT_RDWR)beforeclose()instop_uds(), mirroring the existing SSE listener shutdown contract, plus defensive early-stop guards invmaf_mcp_uds_thread_main.
macOS Homebrew llvm download reliability (build.yml)¶
- Symptom: intermittent network dropouts or CDN blips when downloading large Homebrew bottles (such as
llvm, about 500 MB) on GitHub-hosted macOS runners. The audit of 40 recentbuild.ymlruns found 0 failures, so the baseline rate is low, but network flakiness is an established infrastructure failure mode. - Mitigation: the Homebrew installation in
build.ymlandlibvmaf-build-matrix.ymlruns in a 3-attempt retry loop with backoff (10 s, 20 s) and fallbackbrew fetch --retry.HOMEBREW_NO_AUTO_UPDATE=1andHOMEBREW_NO_INSTALL_CLEANUP=1are exported to avoid costly auto-updates and cleanup consuming runner time.
Other master workflow legs¶
The last 40 and 100 master runs across all workflows showed no other flaky test or build failure; the only other master failure was the release-please token permissions, tracked in #1289.