The metric everyone plans capacity with, tested against the work.

GPUTL read 96.0 at 2 requests a second (registered) and 100.0 from load start at saturation, with SMACT at 81.5 (post-hoc). The completed request rates were 2.01 a second over 589 seconds and 25.06 a second over 483 seconds, 12.5 times apart (post-hoc). At 0.2 requests a second the GPU completed requests at 0.8 percent of the saturated rate while GPUTL read 26.7 (registered), 27 percent of the saturated load-start reading (post-hoc).

On these loads GPUTL exceeded SMACT by 1.1 to 1.6 times in the registered rows. Measured from load start, the saturated ratio is 1.23 in repeat 1 and 1.22 in repeat 2 (post-hoc). The gap between the two fields is real and modest. The larger problem is that neither field tells you how much work was done.

In training, GPUTL read 99.5 and 99.3 and SMACT 90.0 and 87.6 for two configurations 26.6 times apart in real tokens per second: about 5.4 times in token slots per second and about 4.9 times in the share of slots holding a real token. In serving, GPUTL read 96.0 at 2 requests a second and 100.0 at saturation, about 12.5 times apart in completed requests a second (post-hoc). Over that step SMACT rose from 64.3 to 81.5, 1.27 times. At low load both fields did rise with traffic, GPUTL from 26.7 at 0.2 requests a second to 96.0 at 2, so what they failed to show was the capacity still unused once the GPU looked busy.

Idle is the case the standard field does get right. A loaded vLLM server with no traffic read 0 on every utilization field while drawing 116 and 117 watts in the two repeats, so a GPU doing nothing is visible without profiling metrics. TENSO is not a work meter either: it follows numeric precision, reading 1.5 for the fp32 run and 19.1 for the bf16 run. The two repeats agree within 1.2 points on every percentage field and within 2 watts on power.

What to collect instead

Your engineers export the profiling fields per workload, and we join them to pod identity and read them next to the throughput the workload reports:

• SMACT and SMOCC. Streaming multiprocessor activity and occupancy. They show idle and busy, as GPUTL does, and on this run they did not measure work either.
• TENSO. Tensor pipe activity. On this run it moved with fp32 against bf16, so it reads precision as much as work.
• DRAMA. Memory bandwidth, which separates compute-bound from memory-bound before anyone right-sizes the wrong axis.
• Throughput. Tokens, samples or requests per second from the workload itself. The only field in this list that counts work.

The profiling fields ship in the DCGM exporter, but turning them on is not free. DCGM_FI_PROF_SM_ACTIVE is commented out of the shipped default-counters.csv, so it takes a config change, and the exporter container needs SYS_ADMIN to read any of the profiling group at all. In a cluster with a security review, granting SYS_ADMIN to a DaemonSet is a conversation rather than a flag, and it is better to have that conversation now than to discover it mid-engagement. Throughput has to be joined in from the workload's own logs or metrics, which is the part that needs an engineer.

Hardware telemetry tells you a GPU did nothing. It does not tell you what that cost. For that, the billing side has to be right first, and the billing side has two traps that catch almost everyone.

Amortization

Commitment-covered resources report $0 unblended cost. A reserved or Savings-Plan-covered GPU node shows as free in a naive cost ranking, so the single most expensive machine in your fleet vanishes from the top of the list. Every figure we produce is amortized: the commitment is spread across the resources that actually consumed it.

Attribution

A CUR 2.0 export at hourly granularity with resource IDs, normalized across the legacy and 2.0 schemas, then joined to the cluster through provider IDs so a line item becomes a node, a node becomes a namespace, and a namespace becomes a team. Whatever will not join is reported as unattributed rather than quietly distributed, because unattributed spend is itself one of the findings.

The tooling

The analysis runs through gpuaudit, a purpose-built CLI rather than a spreadsheet with your name typed into the header. It reads Parquet and CSV, legacy CUR and CUR 2.0, a single file or a whole partitioned export prefix, with the format and dialect detected rather than configured. Map columns are read as maps, which is the detail most often got wrong and the one that fails silently: get it wrong and every tag reads as null, which quietly converts proven findings into likely ones.

Eight detectors run against billing data alone, spanning all three confidence ratings, and two more run when cluster data is supplied. Each returns a finding with its rating attached, or returns nothing rather than inventing a small number. The output is a client-ready document and a machine-readable findings file, so your engineers can check the arithmetic themselves.

A regression suite of more than 700 tests covers it, which is not a vanity number: the failure mode that matters in billing analysis is silent. Nothing raises, no column is missing, the totals just quietly come out weaker or stronger than the truth. Tests are the only thing that catches that class of bug before it reaches your report.

Cluster allocation

The allocation module reads GPU capacity and pod requests through kubectl or the Python client, maps each node to its EC2 instance through spec.providerID, and joins that to per-resource amortized cost. It ships the exact read-only RBAC the collector needs, list on nodes and pods and get on the kube-system namespace and nothing else, and the collector is a standalone shell script your team runs without installing anything, sending us the one file it writes.

This is what makes the free call's most persuasive number real: how many provisioned GPUs have no pod requesting one. If a node advertises eight and nothing asks for any, utilization is zero by definition and no telemetry argument can rescue it.

One H100 80GB (DCGM reports H100 80GB HBM3) on Lambda. One serving model, Qwen2.5-7B-Instruct, with synthetic random prompts. One 0.5B training model on wikitext-2. The untuned configuration was constructed for the test (fp32, padding to 512, batch 4, per-step CPU tokenization), and no claim is made about how common it is. Single GPU, no multi-GPU traffic.

What the run establishes is narrow and worth stating narrowly: on this GPU, the utilization fields did not show how much work was done. In training they read the same for two configurations of one job 26.6 times apart in throughput. In serving, GPU utilization read 96.0 percent at a completed request rate 12.5 times lower than the same GPU reached at saturation (post-hoc). It does not establish that yours are misleading, by how much, or on which workloads. Nobody can tell you that from the outside, and anyone who offers to is selling you a number they made up.

The only way to get your number is to instrument your cluster and read it next to what it produced. That is what the first call is for, and it is free.