NVIDIA wants CFOs to stop asking "how much compute does my dollar buy?" and start asking "how much does each AI response actually cost?" That's the verdict—and it's both the most nakedly self-interested thing the chipmaker has ever proposed and the most operationally honest metric the industry has had offered to it.
The argument, laid out in NVIDIA's latest infrastructure blog, goes like this: FLOPS per dollar measures raw theoretical throughput, while cost per token measures what enterprises actually consume. These are not the same thing. A chip with impressive peak specifications can deliver poor real-world token output due to memory bandwidth constraints, software inefficiencies, or poor utilization rates. When procurement teams optimize for FLOPS, they optimize for a fantasy. When they optimize for cost per token, they optimize for reality.
NVIDIA's framing has an elegant internal logic. Cost per token = (GPU cost per hour) / (tokens delivered per hour). The numerator—the price of hardware—is visible, comparable, and easy to negotiate. But the denominator—token throughput—is where hardware differences compound. This is what NVIDIA calls the "inference iceberg." What sits above water (GPU hourly rates) gets all the attention. What sits below (memory architecture, interconnect bandwidth, software stack optimization) determines whether you sink or swim. NVIDIA's H100 and B200 GPUs happen to excel at everything beneath the surface.
The self-interest here is almost refreshing in its transparency. By shifting the conversation from raw compute pricing to cost per token, NVIDIA converts its architectural advantages—superior memory bandwidth, NVLink interconnect, CUDA ecosystem maturity—into measurable economic value. Suddenly, a cloud instance running AMD MI300X at $2.10/hour looks expensive if it delivers fewer tokens per second than an H100 at $2.50/hour. The FLOPS-per-dollar argument that custom silicon vendors and AMD use to position themselves as cost competitors evaporates. Cost per token makes NVIDIA's chips look cheap in a way that FLOPS per dollar never did.
But here's the uncomfortable truth: the argument is also right. Enterprise CFOs building AI applications don't care about floating-point operations. They care about the line item on their monthly cloud bill. If that bill scales with tokens generated—which is how most inference APIs are priced—then cost per token is the metric that maps to actual operational expense. It's the first infrastructure metric that makes finance teams feel like they understand what they're buying. That's valuable regardless of who benefits.
The question is whether NVIDIA will put numbers behind the thesis. The blog promises "lowest cost per token in the industry" but offers no third-party benchmarks. Enterprise buyers should demand proof before letting NVIDIA reshape their procurement frameworks. When infrastructure vendors define the metrics, the industry gets the outcomes that serve the vendor. That's not a conspiracy—it's just incentives. NVIDIA's cost-per-token framework is compelling enough that it might become the standard regardless. But the industry should verify before it adopts.