vmafx-controller gRPC service¶

vmafx-controller is the distributed platform controller for VMAFX Phase 4b. It is a single Go binary that exposes VMAF scoring and job orchestration over both gRPC (default :9090) and HTTP/JSON (default :8080). Configuration is by VMAFX_* environment variables only.

This page covers the gRPC interface. The HTTP routes are GET /healthz, GET /readyz, GET /metrics and POST /v1/score (cmd/vmafx-controller/main.go); their request and response shapes match the vmafx-server REST adapter. Authentication is in auth.md.

The controller exposes two gRPC services on the same port:

Quick start¶

1. Run the controller locally. It needs a vmaf binary, for example from core/build-cpu.

   VMAFX_VMAF_BINARY=core/build-cpu/tools/vmaf \
   VMAFX_MODEL_DIR=model/ \
   VMAFX_HTTP_ADDR=:8080 \
   VMAFX_GRPC_LISTEN=:9090 \
   VMAFX_DB_PATH=/tmp/vmafx-controller.db \
   VMAFX_SCORING_ROOTS=/media \
   go run ./cmd/vmafx-controller
   

   VMAFX_SCORING_ROOTS names the inputs callers may score; without it every input is refused (scoring roots).

2. Or build and run the container image.

   docker build -f docker/Dockerfile.controller -t vmafx-controller:dev .
   docker run --rm \
       -e VMAFX_VMAF_BINARY=/usr/local/bin/vmaf \
       -e VMAFX_MODEL_DIR=/usr/local/share/vmafx/model \
       -e VMAFX_DB_PATH=/data/vmafx-controller.db \
       -p 8080:8080 -p 9090:9090 \
       vmafx-controller:dev
   

Configuration¶

All settings are environment variables (12-factor); the controller has no CLI flags beyond --version since ADR-1119. Listen addresses are full addresses (:8080), not bare ports.

The table is generated from api/vmafx-platform.toml (API generation). How the authentication variables combine is on auth.md; the controller does not start when they are inconsistent or a configured tenant is invalid (tenant registry).

VmafxScoring service (direct scoring)¶

Retained from Phase 4a for backward compatibility. Clients that already talk to vmafx-server continue to work against vmafx-controller without changes.

service VmafxScoring {
  rpc Score(ScoreRequest)   returns (ScoreResponse);
  rpc ScoreStream(stream ScoreStreamRequest)
      returns (stream ScoreStreamResponse);  // bidirectional, per-frame (ADR-0933)
  rpc Health(HealthRequest) returns (HealthResponse);
}

Proto: proto/vmafx/v1/vmafx.proto, generated from api/vmafx-platform.toml. Generated stubs: gen/go/ (package vmafxv1).

Example: direct score¶

grpcurl -plaintext \
    -d '{"reference":"/data/ref.yuv","distorted":"/data/dis.yuv","model":"vmaf_v0.6.1"}' \
    localhost:9090 vmafx.v1.VmafxScoring/Score

VmafxController service (job queue + node API)¶

Phase 4b.1 distributed orchestration surface. Proto: proto/vmafx/controller/v1/controller.proto, generated from api/vmafx-platform.toml. Generated stubs: gen/go/controller/.

Client API¶

Used by the CLI, MCP server, and future Web UI to submit and track jobs. The full service definition is in api/vmafx-platform.toml.

Every call carries the tenant of its token, and every job it reads or writes belongs to that tenant; the role each call needs and the tenant rules are in Auth gateway.

Submit a job¶

The reference and the distorted input must lie under the tenant's scoring roots, or the call fails with PERMISSION_DENIED; the node checks them again, following symlinks, before it reads them.

grpcurl -plaintext \
    -d '{"scoring":{"reference":"/data/ref.yuv","distorted":"/data/dis.yuv","backend":"cuda"}}' \
    localhost:9090 vmafx.controller.v1.VmafxController/SubmitJob
# → {"jobId": "550e8400-e29b-41d4-a716-446655440000"}

Poll job status¶

grpcurl -plaintext \
    -d '{"jobId":"550e8400-e29b-41d4-a716-446655440000"}' \
    localhost:9090 vmafx.controller.v1.VmafxController/GetJob
# → {"id":"...","status":"COMPLETED","scoring":{...},"assignedNode":"node-abc"}

Cancel a job¶

grpcurl -plaintext \
    -d '{"jobId":"550e8400-e29b-41d4-a716-446655440000"}' \
    localhost:9090 vmafx.controller.v1.VmafxController/CancelJob
# → {"ok":true,"message":"cancellation requested"}

The job becomes CANCELLED at once and stays so. A pending job is never handed out. A running job keeps running on its node until the node's next heartbeat, at most VMAFX_CONTROLLER_HEARTBEAT_INTERVAL (10 s) later: the heartbeat lists the jobs the node runs, the controller answers with those that were cancelled, and the node cancels them, which kills their vmaf processes, and reports them as failed (cancelled by the controller: ...). That report does not change the job's status (ADR-1567).

Node API¶

Used by vmafx-node worker processes to pull and report work (the node RPCs in the table above).

Node lifecycle (vmafx-node implements it when VMAFX_CONTROLLER_ADDR is set; see node.md):

1. On startup, the node calls RegisterNode with its capability (GPU vendor, available backends, concurrency slots). The controller returns a node_id and a session_token.
2. The node calls Heartbeat every ~10 s with the node_id, the session_token and the IDs of the jobs it runs (running_job_ids, at most 64). The answer's cancel_job_ids names those of them that were cancelled; the node stops them. A node that misses heartbeats for 60 s is evicted, and its running jobs return to PENDING ahead of newer work, so another node picks them up. A node that comes back registers again; if it still finishes such a job, the first final result the controller receives is kept.
3. When the node has capacity, it calls PullWork. The controller assigns the oldest PENDING job whose backend requirement matches the node's capabilities.
4. After the job completes (or fails), the node calls ReportResult with final=true.

Job lifecycle¶

Backend capability matching¶

A job's scoring.backend field specifies which backend the job requires (e.g. "cuda", "sycl", "cpu"). If empty, any node can accept the job. A node must list the required backend in its capability.backends to receive the job.

Job persistence¶

The controller keeps jobs and node sessions in one of two backends, chosen with VMAFX_STORE_BACKEND:

PostgreSQL backend¶

export VMAFX_DB_DSN='postgres://vmafx:secret@db.example:5432/vmafx?sslmode=verify-full'
vmafx-controller migrate                       # schema of the store and of River
VMAFX_STORE_BACKEND=postgres vmafx-controller  # every replica the same way

• Schema. vmafx-controller migrate applies the store's migrations and River's, then exits; run it once per release, before the controllers of that release start. A controller whose database is below the schema it needs reports not ready on /readyz and logs both versions. A controller that starts before its database answers, or before River's tables exist, does not exit: it retries starting River every 2 s (for up to 30 minutes) and reports not ready until River runs, so its replicas can start in any order with the database and the migration.
• Node work is leased. PullWork gives a node a job with a lease (VMAFX_STORE_LEASE_TTL, default 60 s); every Heartbeat naming the job renews it. A lease that is not renewed expires: a sweep that runs every VMAFX_STORE_SWEEP_INTERVAL (default 5 s) on one replica returns the job to pending after a backoff (VMAFX_STORE_BACKOFF_BASE 5 s, doubling up to VMAFX_STORE_BACKOFF_MAX 5 min), and fails it after three lost leases.
• One result per job. A result is recorded only for the attempt the reporting session holds; a node that lost its lease and reports late is refused (PERMISSION_DENIED), and the job keeps the result of the attempt that replaced it. A retried report of an accepted result succeeds without a second write.
• Sessions survive controller restarts. The node ID returned by RegisterNode names a session stored in the database (VMAFX_STORE_SESSION_TTL, default 60 s), so a node keeps working through a controller restart or a replica failure without registering again. A node process that restarts registers again; the jobs it was running return to the queue when their leases expire.
• Tenants. Every query names the caller's tenant, and row-level security hides other tenants' rows from the database role the controller uses. That role must not be a superuser (a superuser bypasses row-level security).

Moving from the SQLite queue¶

VMAFX_DB_DSN=... vmafx-controller import-sqlite --from /data/vmafx-controller.db

import-sqlite copies every job of the SQLite queue file (read-only) into the PostgreSQL store: pending and running jobs arrive pending, finished jobs keep their result, and a second run skips what the first copied. Jobs that carry no tenant (a queue run with authentication disabled) are refused unless --empty-tenant <tenant> names the tenant to give them.

SQLite backend¶

On a restart of a controller with the SQLite backend:

• PENDING jobs are reloaded and re-queued in submission order.
• RUNNING jobs are reset to PENDING (their assigned nodes are gone).
• COMPLETED, FAILED, and CANCELLED jobs are retained for audit.

Schema: cmd/vmafx-controller/queue/schema.sql.

Prometheus metrics¶

/metrics exposes the following in Prometheus exposition format. The scoring counters and histogram are the metric set shared with vmafx-server (pkg/observability/observability.go), so they carry the vmafx_server_ prefix; the queue and node metrics carry vmafx_controller_.

There is no cancelled-jobs counter.

Graceful shutdown¶

The controller listens for SIGTERM and SIGINT. On receipt it:

1. Stops accepting new connections.
2. Waits up to 30 seconds for in-flight requests to drain.
3. Exits with code 0.

Jobs remain in the SQLite database; the next controller instance reloads them.

Further reading¶

• ADR-0711 — Phase 4b.1 decision record.
• ADR-0709 — Phase 4b umbrella architecture.
• ADR-0703 — Phase 4a origin (vmafx-server).
• REST adapter — /healthz, /readyz, /metrics, /v1/score request and response shapes.
• Auth gateway — JWT authentication and tenant isolation.
• k8s deployment guide — Helm chart configuration.