Metric reference¶
Every metric family the VMAFx services serve on their Prometheus /metrics page, generated from the one definition the services build their collectors from (pkg/observability/metricdef). Where each component serves the page and how to scrape it is in the observability guide.
Each component also serves the Go runtime and process collectors of the Prometheus client library (go_*, process_*).
Families¶
| Family | Type | Unit | Labels | Served by | Max series | Description |
|---|---|---|---|---|---|---|
vmafx_build_info | gauge | info | version | vmafx-server, vmafx-controller, vmafx-node | 2 | Constant 1, labelled with the build version. |
vmafx_metrics_read_errors_total | counter | reads | source | vmafx-controller, vmafx-node | 3 | Reads of scraped families that failed, by source; the families of a failed read are absent from that scrape. |
vmafx_server_score_requests_total | counter | requests | none | vmafx-server, vmafx-controller | 1 | Total number of Score requests (HTTP + gRPC). |
vmafx_server_score_errors_total | counter | requests | none | vmafx-server, vmafx-controller | 1 | Total number of Score requests that returned an error. |
vmafx_server_score_duration_seconds | histogram | seconds | none | vmafx-server, vmafx-controller | 17 | End-to-end duration of a Score request in seconds. |
vmafx_server_health_requests_total | counter | requests | none | vmafx-server, vmafx-controller | 1 | Total number of Health / healthz requests. |
vmafx_server_ready_requests_total | counter | requests | none | vmafx-server, vmafx-controller | 1 | Total number of readyz requests. |
vmafx_controller_jobs_submitted_total | counter | jobs | tenant | vmafx-controller | 257 | Total number of jobs submitted to the controller queue. |
vmafx_controller_jobs_completed_total | counter | jobs | tenant | vmafx-controller | 257 | Total number of jobs that completed successfully. |
vmafx_controller_jobs_failed_total | counter | jobs | tenant | vmafx-controller | 257 | Total number of jobs that finished with an error. |
vmafx_controller_jobs_cancelled_total | counter | jobs | tenant | vmafx-controller | 257 | Total number of jobs cancelled before they finished. |
vmafx_controller_jobs_requeued_total | counter | jobs | reason | vmafx-controller | 4 | Total number of running jobs returned to the queue since the controller started, by reason. |
vmafx_controller_jobs_pending | gauge | jobs | tenant | vmafx-controller | 257 | Current number of PENDING jobs in the queue. |
vmafx_controller_jobs_running | gauge | jobs | tenant | vmafx-controller | 257 | Current number of RUNNING jobs in the queue. |
vmafx_controller_queue_oldest_job_age_seconds | gauge | seconds | tenant | vmafx-controller | 257 | Age of the oldest PENDING job in the queue; absent while the tenant has none. |
vmafx_controller_nodes_live | gauge | nodes | none | vmafx-controller | 1 | Current number of registered (live) vmafx-node instances. |
vmafx_controller_job_queue_wait_seconds | histogram | seconds | tenant | vmafx-controller | 3855 | Time a job waited in the queue, from submission to its assignment to a node. |
vmafx_controller_job_duration_seconds | histogram | seconds | tenant, outcome | vmafx-controller | 15420 | Time from a job's submission to its terminal state. |
vmafx_quality_score | histogram | score | tenant, model, profile | vmafx-server, vmafx-controller | 634790 | Distribution of the pooled VMAF scores of completed jobs and Score requests. |
vmafx_stream_sessions | gauge | sessions | none | vmafx-server, vmafx-node | 1 | ScoreStream sessions open now. |
vmafx_stream_sessions_finished_total | counter | sessions | outcome | vmafx-server, vmafx-node | 4 | ScoreStream sessions that ended, by outcome. |
vmafx_stream_frames_total | counter | frames | none | vmafx-server, vmafx-node | 1 | Frame pairs received on ScoreStream sessions. |
vmafx_stream_session_duration_seconds | histogram | seconds | none | vmafx-server, vmafx-node | 17 | Duration of a ScoreStream session, from its opening message to its end. |
vmafx_node_info | gauge | info | backend, vendor | vmafx-node | 42 | Constant 1, labelled with the backend the node runs (VMAFX_BACKEND) and its GPU vendor. |
vmafx_node_slots | gauge | slots | none | vmafx-node | 1 | Number of controller jobs the node runs at once (VMAFX_NODE_SLOTS); 0 without a controller. |
vmafx_node_jobs_running | gauge | jobs | none | vmafx-node | 1 | Number of controller jobs the node is running now. |
vmafx_node_jobs_total | counter | jobs | backend, outcome | vmafx-node | 28 | Total number of controller jobs the node finished, by backend and outcome. |
vmafx_node_job_duration_seconds | histogram | seconds | backend | vmafx-node | 105 | Time the node spent running one controller job. |
vmafx_node_device_memory_used_bytes | gauge | bytes | device | vmafx-node | 9 | Device memory in use on each GPU of the node's vendor, read when Prometheus scrapes. |
vmafx_node_device_memory_total_bytes | gauge | bytes | device | vmafx-node | 9 | Device memory of each GPU of the node's vendor. |
Max series is the most series one process can serve for the family: the product of its label bounds, times the bucket series plus _sum and _count for a histogram. Read at scrape time: vmafx_controller_jobs_requeued_total, vmafx_controller_jobs_pending, vmafx_controller_jobs_running, vmafx_controller_queue_oldest_job_age_seconds, vmafx_controller_nodes_live, vmafx_node_device_memory_used_bytes, vmafx_node_device_memory_total_bytes.
Labels¶
A label with a closed value set takes one of its values or other. An open label reports at most its limit of distinct values per process; every later value is merged under other (counts add up, ages take the oldest), so a series is never dropped. An empty value is reported as none.
| Label | Values | Source |
|---|---|---|
version | open, at most 1 | build version the binary reports with --version |
source | queue, device_memory | queue: the controller's job queue; device_memory: the node's GPU memory (nvidia-smi or the amdgpu sysfs files) |
tenant | open, at most 256 | tenant of the job or request (the JWT tid claim); none on vmafx-server, which has no tenants |
reason | node_lost, controller_restart, assign_rollback | node_lost: the node missed its heartbeats and was evicted; controller_restart: the controller restarted with the job running; assign_rollback: an assignment failed after the job left the queue |
outcome | completed, failed, cancelled | terminal state of the job |
model | open, at most 64 | VMAF model name the score came from; the default model when the request named none |
profile | none | scoring profile of the request; none until scoring requests carry a profile (the device profiles of ADR-1880) |
backend | cpu, cuda, sycl, hip, metal, none | compute backend; none when the job named no backend and the node has none configured |
vendor | nvidia, amd, intel, apple, cpu | GPU vendor of the node's backend (cuda nvidia, hip amd, sycl intel, metal apple) |
device | open, at most 8 | index and name of a GPU of the node's vendor, e.g. 0 NVIDIA GeForce RTX 4090 |