FP8 vs BF16 on H100: what a million tokens costs on AWS, Azure and GCP in 2026

Summary. On a single NVIDIA H100 running Llama-3.1-8B, vLLM measured 517.5 output tokens per second with an FP8 KV-cache against 450.3 with BF16 - 14.9% more throughput, published 22 April 2026. Priced against AWS p5.48xlarge on-demand at $55.04/hour, that moves a million output tokens from $4.24 to $3.69. The same workload costs $7.58 on Azure's ND96isr H100 v5 at $98.32/hour, and ₹223 on E2E Networks in Mumbai at ₹362/GPU-hour. Here is the number nobody puts in the headline: switching the same AWS instance from on-demand to a 3-year reservation takes BF16 from $4.24 to $1.83 per million tokens - a 57% cut, more than four times what FP8 delivers. Quantize anyway, because the two stack. But if you are optimising precision before you have fixed your purchase model, you are working on the smaller number.

What a million tokens actually costs

The table below prices one million output tokens on a single H100, using vLLM's measured throughput for Llama-3.1-8B under load (150 requests, concurrency 8, roughly 20k input and 2k output tokens each). Per-GPU rates come from each provider's 8-GPU H100 node divided by eight.

Sources: Vantage for AWS ($55.04/hour on-demand, $26.971 spot, updated 21 June 2026) and Azure ($98.32/hour), CloudZero for the GCP node at roughly $80-$90/hour (April 2026), E2E Networks for the Indian rate. Throughput from the vLLM benchmark.

Three things fall out of that table.

FP8 saves 13.0% on every row, and only 13.0%. The saving is a pure throughput ratio - 450.3 divided by 517.5 - so it does not care which cloud you are on. It is real and it is free, but it is not the lever people expect.

Azure costs 1.79x what AWS costs for the same silicon. A million tokens is $7.58 on Azure BF16 and $4.24 on AWS BF16. No amount of quantization closes that gap: Azure with FP8 ($6.60) is still 56% more expensive than AWS with BF16.

Spot beats precision by a wide margin. AWS spot at BF16 ($2.08) is 44% cheaper than AWS on-demand at FP8 ($3.69). If your inference tolerates interruption, that decision matters more than your dtype.

The benchmark these numbers come from

The throughput figures are not vendor marketing. They come from a validation study published on the vLLM blog on 22 April 2026 by Jonas Kübler and Kailash Budhathoki of AWS with Eldar Kurtić, Lucas Wilkinson, Matthew Bonanni, Michael Goin and Alexandre Marques of Red Hat AI.

The team ran --kv-cache-dtype fp8 on a single H100 with FlashAttention-3, sweeping input lengths from 256 to 125k tokens at concurrency 1, then running a load test at concurrency 8. For Llama-3.1-8B under load:

That is 14.9% higher output throughput, 13.0% faster total runtime and 14.8% lower median inter-token latency. Two effects combine: each token decodes faster because FP8 halves the memory traffic per attention step, and the 2x memory reduction lets the scheduler pack more concurrent requests onto the same card.

At concurrency 1, the fitted inter-token-latency slope for Llama-3.1-8B dropped from 4.37e-05 to 2.37e-05 ms/token - 54% of BF16, close to the theoretical optimum - while the intercept moved only from 6.44 ms to 6.58 ms.

One caution about terminology, because vendors blur it. This benchmark quantizes the KV-cache and the attention computation, not the model weights. Full FP8 weight-and-activation quantization (W8A8) is a separate change, and the vLLM documentation claims a 2x reduction in model memory and up to 1.6x throughput for that path. The two compose. If someone quotes you a 1.6x FP8 speedup and you are only turning on the KV-cache flag, you will get 1.15x and wonder what went wrong.

Why the purchase model moves more money than the precision

Here is the same AWS instance, priced four ways. Rates from Vantage, 21 June 2026.

The last row is the one worth staring at. A 3-year reservation running at 50% utilisation costs more per token than on-demand FP8. The cheapest contract in the table becomes the second-most-expensive option the moment the card sits idle half the day. Erik Peterson, co-founder and CTO of CloudZero, put it this way: "AI experiments in your business are icebergs." The hourly rate is the part above the waterline.

CloudZero's own billing data makes the same point from the other direction: across its customer base in late 2025, AI spending averaged just 2.5% of total cloud spend even as organisations planned 36% budget increases. Much of the gap sits in GPU capacity that never gets tagged as AI cost because it is idle.

The order of operations for an inference budget is therefore: fix utilisation, then fix the purchase model, then quantize. Most teams do it backwards because quantization is the fun part.

Where FP8 stops paying

FP8 is not free everywhere. The vLLM team was specific about the failure modes, and this is the section most FP8 write-ups skip.

The break-even numbers moved a long way in a short time. Before the fixes shipped in vLLM v0.19.1, Llama-3.1-8B broke even at 24,889 tokens and gpt-oss-20b at 741,565 - far beyond any practical context, which is to say FP8 did nothing for that model. After the fixes: 7,010 and 7,659 respectively. If you benchmarked FP8 on vLLM v0.10.2 and concluded it was not worth it, that conclusion has expired.

For gpt-oss-20b, the hybrid-attention case, the cost picture at AWS on-demand is much thinner:

A 4.6% saving. Real, but you would not schedule a migration for it.

The accuracy question, answered with numbers

The reason FP8 stalled for so long is that it was quietly broken at long context on Hopper. On a 128k needle-in-a-haystack task, FP8 accuracy collapsed from 91% (BF16 baseline) to 13%. The cause was imprecise FP32 accumulation in the H100's Tensor Cores when the contraction dimension exceeds roughly 100k - the same hardware behaviour documented in the DeepSeek-V3 Technical Report. A two-level accumulation strategy borrowed from SageAttention2 brought it back to 89%.

With that fixed, the measured accuracy cost of FP8 KV-cache plus FP8 attention, using the worst case of uncalibrated per-tensor scales:

Those are lower bounds. The evaluations used scale = 1.0 with no calibration data and no per-head tuning, which the authors chose precisely because any vLLM user can reproduce it with one flag. Calibrated results only improve.

The vLLM team's conclusion: "FP8 KV-cache quantization is now ready to be the default starting point for many long-context vLLM deployments and hardware environments."

Turning it on

The change is one flag. For a standard decoder:

            vllm serve meta-llama/Llama-3.1-8B --kv-cache-dtype fp8
          

For a hybrid-attention model with small sliding windows, keep those layers in BF16:

            vllm serve gpt-oss-20b \
  --kv-cache-dtype fp8 \
  --kv-cache-dtype-skip-layers sliding_window
          

Then measure your own cost per million tokens rather than trusting this article's:

            # 1. Benchmark both precisions on YOUR traffic shape
vllm bench serve \
  --model meta-llama/Llama-3.1-8B \
  --num-prompts 150 \
  --max-concurrency 8 \
  --random-input-len 20000 \
  --random-output-len 2000

# 2. cost per 1M output tokens = gpu_hourly_rate / (output_tok_per_sec * 3600 / 1e6)
#    AWS p5.48xlarge on-demand: 55.04 / 8 = 6.88 per GPU-hour
#    At 517.5 tok/s: 6.88 / (517.5 * 3600 / 1e6) = $3.69
          

Run it at your real input and output lengths. A workload with 500-token prompts will show FP8 losing, exactly as the break-even table predicts.

India-specific considerations

Indian teams have a third option that rarely appears in AWS-versus-Azure comparisons: domestic GPU clouds billing in rupees, which removes both the currency exposure and the cross-border egress on training data.

E2E Networks lists H100 80GB at ₹362/hour per GPU on-demand and ₹2,896/hour for an 8-GPU node, from data centres in Mumbai and Bangalore, with commitment pricing for three months or more as low as ₹155.90 per GPU-hour. At that committed rate a million output tokens costs ₹84 with FP8 - roughly a fifth of AWS on-demand BF16.

The dollar conversions above are indicative and move with the rate; price your own workload on the day. Two things do not move: latency to Indian users from Mumbai is lower than from us-east-1, and data that never leaves India is a simpler conversation under the Digital Personal Data Protection Act 2023 when your prompts carry personal data. If you are serving Indian consumers and your KV-cache holds their information, the location of the GPU is a compliance question, not only a cost one. Our cloud FinOps managed service treats those two as the same decision.

What the published comparisons get wrong

While researching this piece we hit a problem worth naming: the widely-cited price comparisons disagree with themselves.

CloudZero's April 2026 comparison lists AWS p5.48xlarge at roughly $6.50-$7.00 per GPU-hour in its table, and its own quick answer puts the 8-GPU node at $55-$60 per hour - consistent, and matching Vantage's $55.04. But the FAQ on the same page states AWS at $3.90 per GPU-hour and GCP at $3.00, citing a 44% price cut in June 2025. Those two halves of one article cannot both be right, and $3.90 x 8 = $31.20 is nowhere near the $55.04 that Vantage records as the current us-east-1 rate.

E2E Networks does the same thing: the headline and detailed pricing table say ₹362/hour, while a comparison table further down the same page says ₹249.

We used the figures that reconcile against a second independent source and against the node price. That is the actual lesson for anyone budgeting inference: do not take a per-GPU-hour number from a comparison article, including this one. Take the node price from the provider's own calculator, divide by the GPU count, and multiply by your own measured throughput. Everything else is someone's stale scrape.

For the wider capacity picture behind these rates, see our analysis of AWS EC2 Capacity Blocks and the GPU price rise and of the Rubin buildout's effect on enterprise cloud budgets.

The order of operations

If you run inference on H100s and want the bill down, do these in order.

Measure utilisation first. A card at 50% turns your cheapest contract into your second-most-expensive one, as the reservation table shows. Nothing else you do matters until this is fixed.

Pick the purchase model second. Spot at BF16 ($2.08) beats on-demand at FP8 ($3.69) by 44%. A 3-year reservation at real utilisation beats both. If your workload cannot tolerate interruption, that is a genuine constraint - but confirm it rather than assuming it.

Then quantize. FP8 KV-cache is 13% off every row of the table, it is one flag, and on the validated paths it costs 1-3 accuracy points. Check your context length against the 7k break-even before you bother.

Then look at the model. gpt-oss-20b serves 831.6 tokens/sec at BF16 against Llama-3.1-8B's 450.3 on the same card - 85% more throughput from choosing a different model, which dwarfs everything above. Cost per million tokens is a model decision before it is an infrastructure decision. Our note on selecting an OpenAI tier for GPT-5.6 makes the same argument for hosted APIs, where the arithmetic is identical and the levers are fewer.

The real cost is usually the idle card, not the precision.

FAQ

How eCorpIT can help

eCorpIT builds and runs inference infrastructure for teams that need the cost per token to make sense before the traffic arrives, not after the invoice does. We benchmark your actual traffic shape on your actual models, set the purchase model against measured utilisation, and turn on quantization where the break-even maths supports it. Our senior engineering teams work across AWS, Azure, Google Cloud and Indian GPU clouds, including the data-residency questions that come with serving Indian users. If your inference bill is growing faster than your usage, talk to us about where the money is actually going.

References

1. The State of FP8 KV-Cache and Attention Quantization in vLLM - vLLM Blog, 22 April 2026. Kübler, Kurtić, Wilkinson, Bonanni, Goin, Marques, Budhathoki.

1. p5.48xlarge pricing and specs - Vantage, updated 21 June 2026.

1. ND96isr H100 v5 pricing and specs - Vantage.

1. a3-highgpu-8g pricing and specs - Vantage, updated 20 April 2026.

1. Cloud GPU Pricing Comparison: AWS Vs Azure Vs GCP For AI Workloads (2026) - CloudZero, updated 27 April 2026.

1. NVIDIA H100 GPU Price in India - E2E Networks.

1. FP8 W8A8 quantization - vLLM Documentation.

1. DeepSeek-V3 Technical Report - arXiv:2412.19437.

1. SageAttention2 - arXiv:2411.10958.

1. vLLM brings FP8 inference to the open source community - Red Hat Developer.

1. 2:4 Sparse Llama FP8: SOTA performance for NVIDIA Hopper GPUs - Red Hat Developer.

1. FinOps for AI Overview - FinOps Foundation.

1. AI Costs And Business Efficiency Report - CloudZero.

1. India's data privacy shift: steering the DPDP compliance and readiness - EY India.

Last updated: 15 July 2026.