ADR-2350: The VMAFx platform keeps its state in PostgreSQL, scales on queue depth and generates its platform surfaces from a definition¶

• Status: Accepted (maintainer answers Q-122 to Q-129 and Q-136, 2026-10-07; scope note 2026-10-08, maintainer: the CloudNativePG backups of work package 7 move to work package 9, with object storage)
• Date: 2026-10-07
• Deciders: maintainer
• Tags: cloud, k8s, controller, node, operator, helm, codegen, rc4, supply-chain, security

Context¶

The distributed platform (controller, nodes, operator, scoring server; ADR-0709) works on one machine but cannot scale or recover by itself. On origin/master 18b04cf8b:

Decision Q-112 (2026-10-07) moves the cloud-native core into RC4 and replaces ADR-1119's queue decision: state out of the processes, disposable and horizontally scalable services, queue-driven scaling, events, every platform surface generated from the API definition, and a standalone profile kept for single-binary and development use (issue #2431). ADR-2001 had put containers, the Helm chart, the operator and the controller / node split into RC5; this record moves the platform core into RC4 and leaves the GPU pool arbiter in libgpudispatch (#1253) in RC5. The maintainer's direction was that high availability alone is not the aim: every core cloud-native feature has to be in place before the quality work of later releases (no-reference live scoring, generation loops, live monitors) builds on it, and as much as possible is generated from the definition.

Constraints that shape the design:

• The node API is an authenticated pull protocol. A node registers with a token whose tenant owns the session (ADR-1522, ADR-1563), pulls work and reports results over gRPC, and decodes untrusted media with FFmpeg.
• River executes a job only inside a River client. JobCompleteTx refuses to run outside a worker and the client has no fetch-without-execute call (Research-2351), so a node behind the gRPC API cannot take and complete a River job.
• golusoris first (ADR-1119): every element maps to a golusoris module or a named external component, and a missing golusoris capability is added to golusoris, not worked around here.
• One generator (HISS-19): the RC4 generator (scripts/codegen/vmafx-api.py, ADR-1852; the proto and OpenAPI emitters of ADR-2044 in draft PR #2258).
• Observability belongs to #2430: metrics, dashboards, alerts and the log bridge have one definition there.

Every version and licence below was checked against the upstream release or documentation on 2026-10-07 (Research-2351).

Decision¶

We will run the controller as several stateless replicas over PostgreSQL, keep node work in leased claims on the job table, put every asynchronous step on River, cache through golusoris cache/twotier, keep inputs and outputs in object storage, export results as signed OCI artifacts, publish job events as CloudEvents through a transactional outbox, scale node pools on queue depth with KEDA, and generate the platform's protobuf, CRDs, OpenAPI and Helm values schema from a platform definition read by the RC4 generator. A standalone profile runs the same code on SQLite with an in-process cache.

D1. State and schema¶

• Every durable fact lives in PostgreSQL: jobs, attempts, node sessions, idempotency records, the outbox, River's tables, the artifact index and the score series. A replica's memory holds caches only; any replica can be killed at any time.
• Schema changes are golang-migrate files embedded in the controller (vmafx-controller migrate up), run by a Kubernetes Job that carries both the Helm and the GitOps hook annotations and is safe to run twice. A controller whose database is below the schema version it needs reports not ready and names both versions; it never starts on an old schema.
• Queries are sqlc-generated (golusoris db/sqlc). Every query keeps the tenant in its WHERE clause as today (ADR-1522), and row-level security adds a second wall: each transaction sets vmafx.tenant_id and the policies refuse other tenants' rows; maintenance runs under a separate role.
• PostgreSQL 18 is the chart's default; 16 and 17 are supported for external servers (TimescaleDB 2.30 supports 16 to 18; PostgreSQL 14 ends on 2026-11-12).

D2. Node work: leased claims, River for everything around them (Q-122)¶

• The job row is the record. A node's PullWork, on any replica, claims the oldest pending job of its tenant that its backends can run with SELECT … FOR UPDATE SKIP LOCKED, opens attempt n (the fencing token), and writes the node session and lease_expires_at.
• Node sessions live in the database (token stored as SHA-256). Heartbeat, on any replica, extends the leases of the jobs the node lists, so a controller replica can die without the node registering again.
• A final ReportResult commits only when attempt and session match the running attempt, and writes the job's outbox event in the same transaction. A job therefore gets one terminal result and one completion event however often it was scored; a late report from a fenced attempt is refused, as ErrNotAssigned is today.
• An expired lease (node gone, network split) returns the job to pending with a delay from River's retry policy until the job's attempt limit, then fails it. The sweep is a River periodic job, so River's elector runs it once.
• A submission sends NOTIFY with the backend, so waiting PullWork long polls on every replica return at once; a short poll covers a lost notification.
• River runs every other step, each one retried with backoff and keyed as a unique job by job and step: staging inputs (presigning), ingesting the per-frame report into the series, rendering the report, exporting and signing artifacts, publishing events, delivering webhooks, retention and garbage collection.
• The node protocol keeps its RPCs; the attempt number and lease length are added fields within vmafx.controller.v1.
• This departs on purpose from the wording of Q-112, which named River as the job queue: the queue of node work is the leased claim table, and River runs the retries, schedules and follow-up steps around it (Q-122).

D3. Data model¶

Tenants stay where ADR-1519 put them (VmafxTenant resources or a file); their specification gains a storage scope, a retention period and webhook endpoints. Objects live under tenants/<tenant>/inputs/… and tenants/<tenant>/jobs/<job>/…; exported artifacts under a per-tenant repository of the configured registry.

D4. Standalone profile¶

VMAFX_PROFILE=standalone runs the same controller on one SQLite file (golusoris db/sqlite), River on its riversqlite driver, an L1-only cache, a local bucket directory, artifacts in an OCI image-layout directory, events delivered to webhooks without a bus, and the series in an SQLite table. The interfaces (store, jobs.Client, twotier, storage.Bucket, the artifact and event publishers) are the same in both profiles, and one conformance suite runs every interface test against both. The queries exist once per SQL dialect (sqlc has a PostgreSQL and an SQLite engine); the conformance suite is what keeps the two in step. River marks its SQLite driver experimental; the standalone profile is for single-binary and development use and the suite covers what it relies on.

D5. Cache¶

golusoris cache/twotier: L1 in-process on otter v2, L2 on Valkey through rueidis, singleflight on misses. Valkey (BSD-3-Clause) is the default L2, from the official valkey-io/valkey-helm chart as an optional dependency or an external endpoint. Dragonfly is a tested drop-in through the same client but not the default: BUSL-1.1 until 2030-11-01 forbids its use as an in-memory data store product or as a service, and that would pass to every operator who hosts VMAFx; the documentation states those terms. The end-to-end suite runs the L2 against both. The cache never holds the only copy of anything: it holds digest-addressed model and artifact manifests, tenant registry snapshots, REST idempotency responses and short-lived job status for polling clients (GetJob, StreamJobs, the operator). Losing L2 slows reads and loses nothing.

D6. Object storage, inputs and outputs (Q-128, Q-136)¶

Inputs and outputs are object references (s3://bucket/key; gs:// and az:// once golusoris storage has those backends). The controller resolves a reference only inside the tenant's storage scope (the successor of the scoring roots, ADR-1577) and gives the node presigned URLs per attempt: GET for inputs, PUT for outputs. The node reads and writes HTTPS and holds no storage credential; the controller signs with workload identity. Large client uploads use presigned PUT or tus (golusoris storage/tus). Local paths stay for the standalone profile.

rclone stays, opt-in, for storage without an S3 API or an OCI registry (Q-136). On the input side it is the node storage mode it is today (mount or http-serve), off by default because its FUSE and eBPF modes need privileges (ADR-1593); a node in that mode writes its outputs through the same remote. On the output side an rclone mover copies a finished job's outputs and exported artifacts to the tenant's rclone remote as a River step on the controller side, with the remote's configuration from a Secret; artifacts travel as an OCI image-layout directory, so their digests and signatures still verify. Presigned URLs and OCI registries stay the defaults, and a tenant names its rclone remote explicitly. The end-to-end suite uses SeaweedFS (Apache-2.0) and an rclone local remote; MinIO is archived and AGPL-3.0.

D7. Score time series (Q-124)¶

Per-frame and window scores go into score_frames and score_windows, written by a River job from the per-frame report the node uploads. They are ordinary tables, made hypertables when TimescaleDB is installed; the queries use standard SQL (date_bin), so they run on both, and retention is a River periodic job (drop_chunks on a hypertable, batched deletes otherwise). The chart's default database uses the Apache-2.0 TimescaleDB edition from CloudNativePG's extension image (PostgreSQL 18, Kubernetes 1.35, containerd 2.1). Compression and continuous aggregates are licensed under the Timescale License and stay opt-in for operators who supply such an image; the documentation states the TSL terms. The standalone profile keeps the series in SQLite.

D8. Artifacts (Q-127)¶

Each finished job can be exported as an OCI 1.1 artifact: artifact type application/vnd.vmafx.result.v1+json, the empty config, and layers application/vnd.vmafx.scores.v1+json, application/vnd.vmafx.report.v1+json and application/vnd.vmafx.provenance.v1+json (the WP5 provenance record). A Sigstore bundle (application/vnd.dev.sigstore.bundle.v0.3+json) is attached as a referrer, signed keyless with the workload's identity where Fulcio can verify the cluster's issuer and with a KMS key reached through workload identity otherwise; cosign signs the manifest by its digest. Models are artifacts of type application/vnd.vmafx.model.v1+json and nodes pull them by digest. The platform definition lists every media type, so constants and the reference page are generated. The client is golusoris container/registry (go-containerregistry), extended upstream with artifact push and pull and the referrers API; its keychains give registry access through workload identity. The standalone profile writes an OCI image-layout directory; the rclone mover of D6 copies the same layout to storage without a registry.

D9. Events (Q-123)¶

Every job state change writes an outbox row in its own transaction (golusoris outbox). The drainer runs on one replica (golusoris leader/k8s) and turns rows into River jobs; a River worker publishes a CloudEvent (structured JSON, spec 1.0.2) on NATS JetStream by default or on Kafka. Types are dev.vmafx.job.{submitted,started,completed,failed,cancelled}.v1 and dev.vmafx.artifact.published.v1; the subject is the job ID; the extension tenantid names the tenant and traceparent carries the trace. Delivery is at least once and consumers deduplicate on the event id. Webhooks are per-tenant endpoints in the VmafxTenant specification, delivered by River jobs with retries and an HMAC signature, in the CloudEvents HTTP binding. Event types and their data schemas are in the platform definition.

D10. Scaling and placement (Q-126)¶

• Each node pool is a Deployment with a KEDA ScaledObject whose PostgreSQL scaler counts the pending jobs the pool can run, through a read-only role; minReplicaCount: 0 scales the pool to zero when idle and out on the first submission. The query is generated with the schema.
• Scale-in drains: on SIGTERM a node stops pulling, finishes its attempts within the pool's grace period and releases what is left; a released attempt returns to pending without counting as a failure. The controller raises controller.kubernetes.io/pod-deletion-cost on node pods with running attempts, so idle pods go first (nodes get no right to patch pods).
• The scoring server scales with KEDA on CPU and in-flight requests (the latter a #2430 metric), never below one replica.
• Placement: pools request GPUs through device plugins by default; a pool can use a DRA ResourceClaimTemplate (resource.k8s.io/v1) instead. Node Feature Discovery labels select CPU pools by instruction set. No values file maps models or jobs to devices: a pool asks for one device of a class, the node discovers its devices at start, and libgpudispatch (RC5) arbitrates inside the pod.
• Kubernetes 1.35 is the minimum (1.34 ends on 2026-10-27); kind runs 1.35, 1.36 and 1.37.

D11. Resilience¶

• SubmitJob takes an idempotency key; the same key from the same tenant returns the first job's ID (unique index). REST takes Idempotency-Key through golusoris idempotency with a shared store; VmafxJob uses its resource UID.
• Retries back off through River; PodDisruptionBudgets cover controller, server and node pools; topology spread places controller and server replicas across nodes and zones; the controller runs two replicas by default with a rolling update.
• Readiness means database reachable and schema current; a missing L2 is reported but does not make a replica unready.
• Leader election only for singletons: River elects for its own maintenance and periodic jobs, golusoris leader/k8s for the outbox drainer.

D12. Security¶

• Workload identity, no static keys, for object storage, registries, CloudNativePG backups and KMS.
• mTLS between every in-cluster service (controller, server, nodes, operator, MCP server) with cert-manager certificates from the chart's issuer or a given cluster issuer and trust-manager bundles; certificates are reloaded on rotation. The certificate authenticates the workload, the JWT the tenant (ADR-0794 unchanged).
• External Secrets for every credential workload identity cannot replace (database, Valkey, bus, registries without federation).
• The NetworkPolicies gain the database, cache, bus and registry egress; result artifacts are signed (D8) as the images already are.
• Nodes hold no storage or database credential (D2, D6).

D13. Generation from a platform definition¶

A platform definition, api/vmafx-platform.toml, is read by the RC4 generator next to core/api/vmafx.toml and may refer to its enums, option groups (ScoreOptions) and proto structs (Provenance). New tables:

Outputs and the tools that run after the generator:

• Protobuf under proto/vmafx/ (controller, scoring service, event data), then one buf generate (configuration version 2) and scripts/proto/postprocess_gen_go.py. cmd/vmafx-controller/proto/generate.sh and gen.go are removed; buf breaking against master is part of the drift gate.
• CRD Go types under api/vmafx/v1/ (including VmafxTenant), then controller-gen (pinned as a Go tool) writes the deepcopy file, the CRD YAML into deploy/helm/vmafx/crds/ (the only tree; config/crd/bases/ is removed) and the RBAC roles. A compatibility check refuses a removed field or narrower validation within v1.
• OpenAPI for the REST surfaces under api/openapi/, then oapi-codegen as today; the client generation of #2321 starts from these files.
• deploy/helm/vmafx/values.schema.json (JSON Schema draft-07 keywords, which Helm 3 and 4 both read), values.yaml with its comments, and templates/_config.gen.tpl.
• SQL migrations stay hand-written (an ordered, reviewed history); a test checks that every persisted message field has its column.

test_vmafx_api_generated_current and a CI step that reruns buf, controller-gen and oapi-codegen fail on any difference, each shown failing on a planted defect (a hand edit of a CRD, of the schema and of a generated proto). The platform surfaces version like the scoring contract: additive within v1, a breaking change as v2 side by side.

D14. Cluster prerequisites and what the chart installs (Q-125)¶

Cluster-scoped operators are documented prerequisites with tested versions: CloudNativePG and its Barman Cloud plugin, KEDA, cert-manager and trust-manager, External Secrets Operator, Node Feature Discovery, and the GPU device plugins or DRA drivers. The chart renders only the namespaced objects that use them (Cluster, ObjectStore, ScheduledBackup, ScaledObject, TriggerAuthentication, Certificate, ExternalSecret, ResourceClaimTemplate), each behind a switch, and refuses to render one whose API the cluster does not serve. Namespaced services may come as pinned optional subcharts (Valkey, NATS) or as external endpoints; every external mode (database, cache, bus, object storage, registry) works without them. No subchart from the retired public Bitnami catalogue.

D15. Node pools and the operator (Q-129)¶

Node pools stay Helm-rendered Deployments with their ScaledObjects, which needs no operator and suits GitOps. The operator stays optional and owns the declarative resources: a VmafxJob is submitted through the controller with its UID as the idempotency key and mirrors the generated status (including Cancelled); VmafxModelTraining keeps its role; VmafxNode reports pool status.

D16. Migration from the SQLite queue¶

• vmafx-controller import-sqlite --from <file> copies the old jobs rows: pending and running jobs become pending with attempt 0, finished jobs are kept, tenants are preserved, and a second run skips IDs already present. A controller pointed at a file of the old schema refuses to start and names that command.
• The chart drops controller.persistence; an optional pre-upgrade Job (migration.importSqlite.existingClaim) mounts the old claim and runs the import once. The controller defaults to two replicas with a rolling update.
• VMAFX_DB_PATH gives way to VMAFX_DB_DSN (or the CloudNativePG application Secret mounted as a file) and VMAFX_PROFILE; the migration page shows the values before and after, as the ADR-1589 move did.
• Nodes and clients change nothing; after the move their sessions survive a controller restart.

D17. Observability boundary¶

Issue #2430 owns the metric definition, dashboards, alerts and the log bridge. The platform's metrics are added through that one definition: queue depth by backend and tenant, lease expiries, attempt outcomes, outbox lag, River queue latency, cache hit ratio per tier, artifact exports, series ingest lag. Queue gauges come from the store (one query, emitted by one replica); trace context travels in River job metadata and in the CloudEvents traceparent extension.

Maintainer answers¶

Each question went to the maintainer with a recommendation; every recommendation was taken (ledger Q-122 to Q-129, 2026-10-07); Q-136 extends Q-128 to outputs and artifacts.

Alternatives considered¶

Consequences¶

• Positive: any controller replica or node can die mid-job and the job finishes once; nodes scale with demand and to zero; results, reports, provenance and models leave the cluster as signed, digest-addressed artifacts; integrations subscribe to events instead of polling; nodes need no storage credential and no FUSE privilege; the platform surfaces stop drifting because they are generated.
• Negative: a production installation needs PostgreSQL, the cluster operators of D14 and, optionally, Valkey and a bus; the queries exist in two dialects; golusoris needs the additions listed below before the dependent packages land; the environment contract changes (VMAFX_DB_PATH).
• Follow-ups:
• With this record, ADR-1119, ADR-0711 and ADR-1589 become "Accepted (partially superseded by ADR-2350)" for the queue, the SQLite persistence and the one-replica workload, ADR-0719 and ADR-1526 for rclone as the default input path, and ADR-2001 for its RC4 / RC5 cloud-native rows. The statements of the old decision are rewritten in the same change: cmd/vmafx-controller/AGENTS.md:49-51 (the prohibition to adopt golusoris jobs), cmd/vmafx-controller/main.go:20-23,233, deploy/helm/vmafx/AGENTS.md:145-146, deploy/helm/vmafx/templates/controller.yaml:5-9,39, deploy/helm/vmafx/values.yaml:53-54,69, deploy/helm/vmafx/values.schema.json:33, docker/Dockerfile.controller:31,182, docs/development/k8s-deployment.md:162,181-182,436, docs/server/controller.md:31,46,64,204-213,246 and docs/architecture/phase4b-distributed-platform.md:33; until the store lands they describe the SQLite queue as the current, transitional state.
• cmd/vmafx-operator/AGENTS.md invariant 5 (a hand-added FinalScore field) and cmd/vmafx-controller/AGENTS.md fx item 6 (hand-written bindings) no longer match the tree and go with the generation package.
• golusoris additions, each filed upstream with a reproduction: River driver choice in jobs (SQLite) and the River update; an L1-only mode and cross-replica invalidation in cache/twotier; a shared store and a gRPC interceptor in idempotency; GCS and Azure Blob backends and presigned PUT in storage; artifact push, pull and referrers in container/registry; client-certificate verification and reload in grpc (keepalive is golusoris#589); CloudEvents in pubsub and outbox; edition detection in db/timescale; an SQLite source in db/migrate.
• Operator documentation under docs/cloud/ grows with each package (prerequisites and tested versions, profiles, backups and recovery, scaling, security, migration from the SQLite queue).
• docs/roadmap.md and the bodies of #1251, #1252 and #1253 move the platform core from RC5 to RC4.

Work packages¶

One pull request each, through the merge train, in this order. Each carries failing-first tests, its operator documentation and, where it changes the platform's behaviour, a kind end-to-end case and a standalone-profile case.

Packages 2 to 4 also cover the generated surfaces the planning audit found written by hand (protobuf, CRDs including VmafxTenant, the duplicated CRD trees, the Helm values schema). Bindings in further languages (Swift, Kotlin, TypeScript) belong to the C API and stay with their 1.1 and 1.2 issues.

Supply-chain impact¶

• New runtime dependencies (through golusoris modules, pinned in go.mod): River (MPL-2.0), pgx (MIT), rueidis (Apache-2.0), aws-sdk-go-v2 (Apache-2.0), go-containerregistry (Apache-2.0), sigstore-go (Apache-2.0), nats.go (Apache-2.0) and franz-go (BSD-3-Clause). Build and test: sqlc (MIT), controller-tools (Apache-2.0), buf (Apache-2.0), SeaweedFS (Apache-2.0) in the suite.
• Cluster prerequisites with tested versions in docs/cloud/: CloudNativePG 1.30.1 and the Barman Cloud plugin 0.15.1, KEDA 2.21.0, cert-manager 1.21.2, trust-manager 0.25.0, External Secrets Operator 2.12.0, Node Feature Discovery 0.19.0, GPU device plugins or DRA drivers. Images (PostgreSQL with the TimescaleDB extension image, Valkey, NATS) and chart dependencies are pinned by version and digest.
• Attack surface: three network services (database, cache, bus) behind NetworkPolicies and TLS; nodes lose their storage credentials and, by default, the FUSE and eBPF privileges.

References¶

• Q (Q-112, 2026-10-07): "Re-plan + build in RC4, all core cloud-native features: state out of the processes (Postgres via CNPG with backups/PITR, River queue, two-tier cache with Valkey/Redis L2, S3-compatible object storage, OCI artifact export via ORAS, TimescaleDB score time series); disposable horizontally scalable services; KEDA queue-driven scaling incl. scale-to-zero GPU nodes, DRA/NFD placement; outbox + CloudEvents on NATS/Kafka; CRDs/proto/OpenAPI/Helm schema generated from the RC4 definition, migrations as jobs; idempotency, retries, PDBs, topology spread; workload identity, mTLS, External Secrets; standalone SQLite profile kept. Supersedes ADR-1119's queue decision."
• Q (Q-122): "Leased claims on the job table (SKIP LOCKED, attempt number as fencing token, sessions in DB) over the controller's gRPC API; River runs retries, schedules and follow-ups around them; no DB credentials on nodes"
• Q (Q-123): "NATS JetStream default, Kafka supported and tested"
• Q (Q-124): "Apache edition by default, plain tables when the extension is missing, TSL features opt-in"
• Q (Q-125): "Documented prerequisites; the chart renders their custom resources; no operator installed by our chart"
• Q (Q-126): "Device plugins + NFD labels by default; DRA opt-in per node pool"
• Q (Q-127): "Extend golusoris container/registry (push/pull, referrers) upstream; own application/vnd.vmafx.* media types incl. models; cosign signs by digest"
• Q (Q-128): "Presigned object URLs by default (no storage credentials on nodes); rclone mounts opt-in"
• Q (Q-129): "Helm Deployments per pool with KEDA ScaledObjects on queue depth (scale to zero)"
• Q (Q-136): "Keep rclone as an opt-in mover for outputs and artifacts too (backends without S3 or OCI), besides opt-in input mounts"
• Maintainer question on the cache tiers (2026-10-07), answered with the recommendation: L1 golusoris cache/memory on otter v2; L2 Valkey by default; Dragonfly a tested drop-in, not the default, because of BUSL-1.1; both tested in the end-to-end suite; the documentation states the Dragonfly terms.
• Issues #2431, #2430, #2155, #2321, #1251, #1252, #1253.
• Research-2351 (versions, licences and limits checked on 2026-10-07).
• ADR-1119, ADR-0711, ADR-1589, ADR-2001, ADR-0709, ADR-0719, ADR-1526, ADR-1522, ADR-1519, ADR-1518, ADR-1563, ADR-1577, ADR-1593, ADR-0794, ADR-1852; ADR-2044 (draft PR #2258).