You are paying for GPUs that never did the work.
Your utilization panel reports that the GPU was busy, not how much work it did. On an H100 we ran one training job two ways, one of them deliberately set up badly. They read 99.5 and 99.3 percent utilization, and one processed 26.6 times as many real tokens per second. We join your AWS bill to what your cluster says each GPU was doing, rank what is recoverable, and show the evidence behind every dollar.
Same utilization reading, 26.6 times the real tokens per second: the tuned run moved about 5.4 times as many token slots a second, and almost all of them held real tokens, against about one in five for the untuned run. Standard GPU utilization (DCGM_FI_DEV_GPU_UTIL) reports the share of each sample in which any kernel was running, not how much of the GPU that kernel used or how much of its work was real tokens rather than padding. Readings are registered DCGM means from repeat 1; throughput is post-hoc, from the training log. One H100, one 0.5B model on wikitext-2, and an untuned configuration built for the test, so this says nothing about how common it is. Every row and the raw logs are in the full measurement record.
The same H100 also served Qwen2.5-7B-Instruct on vLLM with DCGM sampled at 1 Hz; the full measurement record is on /method.
Nobody in your org can answer what fraction of the GPU bill produced work.
Not because your team is careless. Because answering it needs two datasets joined that almost nobody has both halves of, and the number everyone reaches for instead is measuring something else entirely.
The metric is the wrong metric
DCGM_FI_DEV_GPU_UTIL is a coarse duty-cycle metric: it reports the share of each sample in which any kernel was running. It says nothing about throughput, the work actually done, and nothing about cost until the bill's line_item_resource_id is joined to each node's spec.providerID. Two configurations of one training job on an H100, one of them set up badly on purpose, read 99.5 and 99.3 percent while one processed 26.6 times the real tokens per second of the other. Telemetry can read 99 percent while those GPU hours go to padded sequences or small batches.
The bill hides the expensive machines
Commitment-covered resources report $0 unblended, so your most expensive GPU node vanishes from a naive cost ranking. Untagged spend has no owner to ask. Amortized cost per resource is the only view that survives contact with finance. The bill shows what each instance hour and commitment cost, not which namespace or workload used it.
So capacity planning is a guess
Every request for more GPUs gets approved against a utilization number that read the same for two configurations 26.6 times apart in throughput, and no one in the room has the instrument to say so. On a node without MIG or time-slicing, a pod that requests nvidia.com/gpu holds each GPU it asks for whether it works or idles, and GPUs no pod requests are still paid for in the node's price. The gap compounds every quarter the fleet grows.
Read what the hardware did. Join it to the bill. Recover what the evidence supports.
Measure
Your engineers run our read-only collector and, where you have them, export DCGM profiling metrics: multiprocessor activity, tensor pipe activity, memory bandwidth. We read them next to the throughput your workloads already log, because no utilization field on its own measures work. If yours has never had this turned on, step one is a written spec your team uses to turn it on.
Attribute
A CUR 2.0 export at hourly granularity with resource IDs, normalized and amortized, then joined to the cluster through provider IDs. The output is cost per GPU resource, per team, per namespace, per model, and the share of it that carries no owner at all.
Recover
A remediation sequence ordered by dollars per unit of disruption. Requests right-sized against measured p95 plus headroom, time-slicing or MIG where workloads are small enough to share, bin-packing and autoscaler tuning so scale-in actually happens, idle reclamation with owner notification. Your engineers apply each change in your own change windows; we write the spec and check the result against the bill.
Every number we hand you carries its evidence.
The industry norm is one confident figure that does not survive its first contact with a skeptical staff engineer. Ours is lower, and it is built to hold up in the room.
Proven is a state a finding reaches, not a property of your bill. What sets the size of that column is how much of the billing period the evidence covers. A cluster read once on the day of the call converts almost none of it. On the specimen's cluster, a week of sampling moves $14,101 a month of the cluster finding into PROVEN, and a full month moves $60,432, from the identical export. That is why the collector is built to be left running through a billing period rather than run once.
Unproven dollars are excluded from the recoverable total, and the exclusion is enforced in the tooling rather than promised in a footnote. Under client pressure, discipline is the thing that bends. Code does not.
This is a narrow practice, on purpose.
Worth a conversation
- Above $50,000 a month in GPU spend. That is the gate. Total cloud spend and headcount are secondary.
- Training or inference on Kubernetes on AWS: EKS or self-managed on EC2. On Google Cloud? Ask us first.
- No dedicated FinOps function, or one person doing it alongside another job.
- An engineering leader who can approve a five-figure invoice without a procurement cycle.
Not worth either of our time
- Under $20,000 a month in total cloud spend. The fee would not clear the finding.
- Fully managed inference only, such as Bedrock or serverless SageMaker. There is no cluster to instrument.
- A team that already joins each workload's own throughput, alongside its SM, tensor and DRAM activity, to the bill, node by node. You have done the hard part. A team scraping only
DCGM_FI_DEV_GPU_UTILhas measured occupancy, not waste, and is worth a conversation. - Anyone whose opening question is what percentage of savings we take.
Four ways in. Fixed fee, every one.
Never a percentage of savings. Contingency pricing gives the person measuring your infrastructure a financial stake in the size of the number, which is the one incentive you do not want in the room.
Spend Snapshot
Three numbers, live on one call: amortized GPU spend per resource for the last full month, the share carrying no owner tag, and how many provisioned GPUs have zero pods requesting one. That third number needs no new instrumentation. Your engineer runs our collection script and sends the one file it writes, plain JSON you can read before sending, and we sign a mutual NDA first if you ask.
GPU Waste Diagnostic
Twelve to eighteen pages. Cost per GPU resource, team and namespace. Ranked recoverable spend with a dollar figure and a confidence rating on every line, the evidence gaps named explicitly, and the instrumentation plan if you do not have telemetry yet. Where an environment needs stakeholder interviews, per-model attribution and a remediation plan built with your team, that is a two-week report at $15,000 to $25,000. Read a specimen report before you buy one.
Recovery Sprint
Priced from the findings of the diagnostic, scoped to the specific items named in the SOW. A written spec your team applies, change by change: telemetry, requests right-sized against measured p95, GPU sharing where it fits, autoscaler settings so scale-in happens, idle reclamation with owner notification. We check each change against the next bill.
Cost Watch
Right-sizing that runs once at deployment is not right-sizing. Quarterly attribution against the same baseline, drift analysis, alerting on new unowned GPU spend and on allocation without utilization, commitment review, standing engineering channel. Two-quarter minimum. Notice given in the first quarter still leaves the second quarter invoiced; after that, notice given at least 30 days before a quarter ends stops the next quarter's invoice. A prepaid quarter runs to its end.
Forty-five minutes and a billing export is enough to know whether there is anything here.
One month of billing data and fifteen minutes describing your cluster. If the numbers come back clean, we will say so on the call and that is the end of it.
- Direct
- 301-802-1073
- Response time
- Same business day