Dashboard tour¶

VMAFx ships ten Grafana dashboards, generated from the same metric definition the services use (deploy/grafana/dashboards/; the Helm chart and the Compose example install them). This page says what each one is for and how to read it. Every dashboard has the same three variables at the top: Prometheus data source, Job and Instance (the scrape jobs and targets that serve vmafx_build_info; All selects every one). Dashboards about tenants add Tenant, the tenants that submitted jobs. The Deploys and restarts annotation marks each component start with its version, so a step in a graph can be read against a rollout. The links at the top right open the other VMAFx dashboards with the same time range and variables.

Overview¶

vmafx-overview.json answers, row by row:

A growing Oldest pending job with Live nodes above zero and low Node slot use means no node can take the queued work: compare the jobs' backend with the nodes' (vmafx_node_info).

When something looks wrong: Components up below the number of pods is component down; pending jobs with no live node is no live nodes; an old pending job is queue aging.

Quality¶

vmafx-quality.json follows the scores the platform produced. Its Tenant, Model and Profile variables narrow the selection.

A drop on one model only after a model change points at the model; a drop on one tenant only points at its content or encodes.

The alert behind this dashboard is score regression.

Nodes and devices¶

vmafx-nodes.json covers the worker nodes; its Job and Instance variables list the node targets (series vmafx_node_info), Backend the backends.

When something looks wrong: GPU memory missing on a GPU node, with Device memory reads failing above zero, is metrics read errors.

Live sessions¶

vmafx-live.json covers ScoreStream, the per-frame streaming path of vmafx-server and vmafx-node (ADR-0933).

A session the client stops (cancelled call or passed deadline) counts as cancelled, not failed.

SLO report¶

vmafx-slo.json reports each SLO over the dashboard's time range, 30 days by default. The objectives are the rule settings (monitoring.slo), which the rules record as vmafx:slo_objective{slo}; the events are recorded every 5 minutes per SLO as vmafx:slo_events:rate5m and vmafx:slo_bad_events:rate5m, and the report adds them up over its window, so every 5 minutes count by their traffic.

A burn rate of 1 spends the budget exactly over 30 days; the fast burn-rate alert fires at 14.4 by default.

The burn-rate alerts behind this report: job error budget, Score error budget and Score latency budget. The objectives are set in monitoring.slo (Kubernetes, Compose).

Usage and cost¶

vmafx-usage.json counts what each tenant used over the time range, 7 days by default, and prices it (decision Q-193):

The cost model is the rule settings' monitoring.cost: a price per second of job run time (perJobSecond) and one per job (perJob), the same for every backend, in currency (decisions Q-193, Q-208). The per-job price applies to completed and failed jobs; cancelled jobs are not charged. The run time of every finished job counts, whatever its outcome; it is the controller's vmafx_controller_job_duration_seconds. A promtool case in deploy/prometheus/vmafx-rules.test.yaml evaluates the per-job cost query: a cancelled job adds no cost. The prices are recorded as vmafx:price_job_second and vmafx:price_job, only while they are above 0: an unset price leaves its cost panels empty, and no price is ever assumed.

Set the prices in monitoring.cost (Kubernetes, Compose); they are yours to choose.

Capacity¶

vmafx-capacity.json sets the nodes' measured capacity against the demand:

Capacity is the nodes' slots times 3600 over the mean job run time of the last 6 hours (vmafx_node_job_duration_seconds), demand the jobs submitted in the last hour; the forecasts are predict_linear over the last 6 hours (queue) and the last day (demand), the growth deriv over the last day.

Headroom below zero means the queue grows: the queue aging alert follows. Add nodes or slots (VMAFX_NODE_SLOTS) for the backend the jobs ask for.

GPU exporter dashboards¶

GPU utilisation, encoder and decoder load and power come from the vendors' exporters, not from VMAFx. One dashboard per exporter is generated, each with its own Job and Instance variables over the exporter's targets:

Import one only where its exporter runs. The contract test allows an exporter's series only on the dashboard tagged vmafx-exporter-<name>, so no VMAFx dashboard shows "No data" on a cluster without that exporter.

See also¶

• Alert runbooks: what to do when an alert fires.
• Metric reference: every series the dashboards read.
• Monitoring on Kubernetes and the Compose stack: installing the dashboards.
• Observability (developer guide): how the dashboards are generated and checked.