TL;DR
- Match VRAM to the model, then buy bandwidth: RTX 5090 for 7-14B, L40S or A6000 for 32B, H100 or H200 for 70B and up.
- Budget about 2.4 GB per billion parameters in FP16 and 0.62 GB in 4-bit, then add KV-cache for context times concurrency.
- On PowerGPU an RTX 5090 at $0.439 sustains roughly 2,800 tokens per second on 8B; interruptible capacity halves the cost per token.
- Cost per million tokens is hourly rate divided by tokens per second times 3,600, times one million; measure it at real concurrency.
Three rules before any benchmark
- VRAM decides what runs. Weights plus KV-cache must fit, with headroom: ~2.4 GB per billion parameters in FP16, ~0.62 GB in 4-bit, then add the cache for your context × concurrency. The VRAM guide has the full tables.
- Bandwidth decides how fast. Decoding is memory-bound: tokens per second track GB/s more than TFLOPS. That is why a 1.79 TB/s RTX 5090 beats a 1 TB/s RTX 4090 by more than its FLOPS suggest, and why HBM cards dominate 70B serving.
- Batch decides the bill. Continuous batching (vLLM, SGLang, TGI) turns one card into a dozen concurrent streams. Cost per million tokens is only meaningful at the concurrency you will actually run.
The table: model size → card → $ per million tokens
Indicative aggregate throughput under vLLM continuous batching (roughly 1k tokens in, 1k out, batch 32–64) and the resulting cost at today's on-demand rates — snapshot 2026-09-14. Treat the tokens/s column as order-of-magnitude community numbers and benchmark your own model; the price column is exact.
| Model | Precision / weights | Card | On-demand | ~tokens/s | $ / M tokens | Note |
|---|---|---|---|---|---|---|
| Llama 3.1 8B | FP16, 16 GB | RTX 4090 24 GB | $0.327 | ~2,200 | $0.041 | The default: fits with KV-cache room on 24 GB. |
| Llama 3.1 8B | FP16, 16 GB | RTX 5090 32 GB | $0.439 | ~2,800 | $0.044 | 78% more bandwidth than the 4090 shows up directly in tokens/s. |
| Llama 3.1 8B | 4-bit, ~6 GB | RTX 3060 12 GB | $0.042 | ~900 | $0.013 | The cheapest working endpoint on the sheet. |
| Qwen 2.5 14B | FP16, 28 GB | RTX 5090 32 GB | $0.439 | ~1,500 | $0.081 | 32 GB holds FP16 14B with modest context. |
| Qwen 2.5 32B | 4-bit, ~20 GB | RTX 4090 24 GB | $0.327 | ~700 | $0.130 | AWQ/GPTQ 4-bit on 24 GB; quality within a point of FP16. |
| Qwen 2.5 32B | FP16, 64 GB | H100 PCIE 80 GB | $1.867 | ~2,400 | $0.216 | FP8 on Hopper roughly doubles this again. |
| Llama 3.1 70B | 4-bit, ~40 GB | RTX A6000 48 GB | $0.281 | ~350 | $0.223 | Cheapest single card that holds a 70B. |
| Llama 3.1 70B | 4-bit, ~40 GB | H100 PCIE 80 GB | $1.867 | ~1,200 | $0.432 | One 80 GB card, real batch sizes, FP8 KV-cache. |
| Llama 3.1 70B | FP8, 70 GB | H200 141 GB | $2.791 | ~2,000 | $0.388 | 141 GB leaves room for long contexts and big batches. |
| Llama 3.1 405B | FP8, ~405 GB | H200 141 GB | $2.791 | ~600 | $1.292 | Tensor-parallel across 4× H200 — figure is per GPU. |
Interruptible capacity halves every number in the price column. Formula, if you want to redo it with your own measurement: $/M = $/hr ÷ (tok/s × 3600) × 1,000,000.
7B–14B: consumer cards win
Below 16 GB of weights nothing beats GeForce silicon per dollar. The RTX 5090 ($0.439/hr, 32 GB GDDR7) is the current king of tokens per dollar for 8B FP16 and 14B models; the RTX 4090 ($0.327/hr) is a hair slower and cheaper; the RTX 3090 ($0.108/hr) is the budget pick when throughput matters less than the hourly rate. For always-on endpoints that must survive months of 24/7 duty, the datacenter L4 ($0.225/hr, 72 W) and L40S trade a little speed for passive cooling and ECC.
24B–32B: the 48 GB middle
A 32B model is ~64 GB in FP16 and ~20 GB in 4-bit. Quantized, it runs on a 24 GB card with short contexts; for serious concurrency you want 48 GB. The L40S ($0.514/hr) is the datacenter answer — FP8, 864 GB/s, built for 24/7 — and the RTX A6000 ($0.281/hr) the cheapest 48 GB on the sheet. Compare them in RTX 5090 vs L40S and RTX 6000 Ada vs A6000.
70B and up: HBM or nothing
Seventy billion parameters is ~40 GB in 4-bit and ~140 GB in FP16. One 80 GB card serves the quantized model with real batch sizes: the H100 PCIe ($1.867/hr) with FP8 KV-cache, or the A100 PCIe ($0.374/hr) when budget wins. Full-precision 70B, 100B+ models and long-context serving belong on the H200 ($2.791/hr, 141 GB, 4.8 TB/s) or tensor-parallel across 2–8× H100 SXM on one machine — the per-GPU price does not change with count. Frontier-scale inference with FP4 is where the B200 ($5.425/hr) earns its rate: several times the H100's tokens per second at under 3× the price.
Serving patterns that change the bill
- Quantize first. AWQ/GPTQ 4-bit halves the card class you need and rarely hurts chat quality. FP8 on Hopper/Ada/Blackwell is nearly free quality-wise and 1.5–2× faster.
- Reserve the baseline, burst serverless. A reserved card at −35% carries steady traffic; serverless workers absorb spikes and scale to zero.
- Mount the model library read-only. New workers skip the 40 GB download; cold starts become load-to-VRAM.
- Prefix caching for agents. If prompts share a long system prefix, SGLang can double effective throughput over plain vLLM.
- Measure, then commit. Ten minutes on two candidate cards costs cents; the vLLM guide ships the exact benchmark command.
Put the numbers to work
Every price in this guide is our live rate — fixed, ≥30% under the market median, billed per second. Deploy the exact setup above from the console in about 30 seconds, paid in crypto, no card and no KYC.
