The metric everyone plans capacity with, tested against the die.

DCGM_FI_DEV_GPU_UTIL reports one thing: was any kernel resident on the device during the sample window. It says nothing about how much of the device that kernel used.

A Tesla T4 has forty streaming multiprocessors. A 64-element add occupies one of them. Launched in a tight loop it keeps something resident nearly every sample window, so the field reports a healthy-looking number while thirty-nine fortieths of the chip idles and the tensor cores never fire at all.

This is not a misconfiguration and it is not unique to any vendor's dashboard. The field is doing exactly what it documents. The problem is that it sits on the GPU utilization panel of nearly every default Grafana board, so it becomes the number in the capacity plan, the number in the quarterly review, and the number cited the last time somebody approved a fleet expansion.

What we deploy instead

The profiling fields, exported per workload and joined to pod identity:

• SMACT — the fraction of streaming multiprocessors with work resident. The honest denominator.
• SMOCC — warp occupancy within those multiprocessors. Distinguishes busy from saturated.
• TENSO — tensor pipe activity. Zero here on a GPU you bought for tensor throughput is a finding on its own.
• DRAMA — memory bandwidth utilization, which separates compute-bound from memory-bound before anyone right-sizes the wrong axis.

These are not exotic. They ship in the DCGM exporter and cost nothing to turn on. The reason most clusters do not have them is that nobody was asked the question that requires them.

Hardware telemetry tells you a GPU did nothing. It does not tell you what that cost. For that, the billing side has to be right first, and the billing side has two traps that catch almost everyone.

Amortization

Commitment-covered resources report $0 unblended cost. A reserved or Savings-Plan-covered GPU node shows as free in a naive cost ranking, so the single most expensive machine in your fleet vanishes from the top of the list. Every figure we produce is amortized: the commitment is spread across the resources that actually consumed it.

Attribution

A CUR 2.0 export at hourly granularity with resource IDs, normalized across the legacy and 2.0 schemas, then joined to the cluster through provider IDs so a line item becomes a node, a node becomes a namespace, and a namespace becomes a team. Whatever will not join is reported as unattributed rather than quietly distributed, because unattributed spend is itself one of the findings.

The tooling

The analysis runs through gpuaudit, a purpose-built CLI rather than a spreadsheet with your name typed into the header. It reads Parquet and CSV, legacy CUR and CUR 2.0, a single file or a whole partitioned export prefix, with the format and dialect detected rather than configured. Map columns are read as maps, which is the detail most often got wrong and the one that fails silently: get it wrong and every tag reads as null, which quietly converts proven findings into likely ones.

Eight detectors run against billing data alone, spanning all three confidence ratings, and two more run when cluster data is supplied. Each returns a finding with its rating attached, or returns nothing rather than inventing a small number. The output is a client-ready document and a machine-readable findings file, so your engineers can check the arithmetic themselves.

Cluster allocation

The allocation module reads GPU capacity and pod requests through kubectl or the Python client, maps each node to its EC2 instance through spec.providerID, and joins that to per-resource amortized cost. It ships the exact read-only RBAC you would grant, in a minimal and a standard version, plus a standalone shell script your team can run themselves without installing anything.

This is what makes the free call's most persuasive number real: how many provisioned GPUs have no pod requesting one. If a node advertises eight and nothing asks for any, utilization is zero by definition and no telemetry argument can rescue it.

One GPU model, two workloads chosen to bracket the range, one machine. A T4 is not an H100 and the ratio on your hardware will be different. These are synthetic phases from a controlled run, not a production trace.

What the run establishes is narrow and worth stating narrowly: the metric almost everyone plans capacity with can be wrong by two orders of magnitude, and it fails silently. It does not establish that yours is wrong, by how much, or on which workloads. Nobody can tell you that from the outside, and anyone who offers to is selling you a number they made up.

The only way to get your number is to instrument your cluster and read it. That is what the first call is for, and it is free.