TL;DR
- On PowerGPU one vLLM template instance gives an OpenAI-compatible endpoint; 8B on an RTX 5090 costs $0.439 per hour.
- Expect roughly 2,800 aggregate tokens per second for 8B on an RTX 5090, 900 for 32B 4-bit, 550 for 70B 4-bit.
- Cost per million output tokens equals hourly rate divided by tokens per second times 3,600, times one million; reserved cuts 35 percent.
- The commonest OOM is leaving max-model-len at the model's 128k maximum; cap context and use an fp8 KV-cache.
Deploy in one command
powergpu launch --gpu rtx-5090 --template vllm --disk 60 \
--env MODEL=meta-llama/Llama-3.1-8B-Instruct
# ✓ i-52ab77c1 · https://i-52ab77c1.powergpu.ai:8000/v1
curl https://i-52ab77c1.powergpu.ai:8000/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{"model":"meta-llama/Llama-3.1-8B-Instruct",
"messages":[{"role":"user","content":"ping"}]}'The template downloads weights on first boot (billed bandwidth: a 16 GB model costs $0.16). Put HF_HOME on a volume and every future instance skips the download.
Pick the card for the model
| Model class | Card | $/hr | Aggregate tok/s ≈ | $ / M output tokens ≈ |
|---|---|---|---|---|
| 7–8B FP16 | RTX 5090 | $0.439 | 2,800 | $0.044 |
| 14B FP16 / 32B-4bit | L40S | $0.514 | 900 | $0.159 |
| 70B-4bit | H100 PCIE | $1.867 | 550 | $0.94 |
Throughputs are typical continuous-batching aggregates at moderate context; your prompt mix will move them ±40%. That is why the benchmark section exists.
The flags that matter
--max-model-len 8192 # cap context = cap KV-cache = predictable memory
--gpu-memory-utilization 0.92 # default 0.90; raise carefully once stable
--max-num-batched-tokens 8192 # throughput/latency dial — higher = more tok/s
--kv-cache-dtype fp8 # halves cache on Hopper/Blackwell — free VRAM
--tensor-parallel-size 2 # split across 2 GPUs when one is shortEverything else can stay default until the benchmark says otherwise. The single most common OOM cause is leaving --max-model-len at a model's 128k maximum "just in case" — the KV-cache section shows what that costs.
Benchmark before you believe
vllm bench serve \
--base-url https://i-52ab77c1.powergpu.ai:8000 \
--model meta-llama/Llama-3.1-8B-Instruct \
--dataset-name random --random-input-len 512 --random-output-len 256 \
--request-rate 16 --num-prompts 512
# → throughput, TTFT p50/p99, ITL p50/p99Judge two numbers: aggregate tok/s (your cost) and p99 time-to-first-token (your users). Raise --request-rate until p99 TTFT crosses your budget — that rate is the card's honest capacity for your traffic shape.
Price your tokens
$/M output tokens = price_per_hour / (tokens_per_sec x 3600) x 1e6
# benchmark said 2,910 tok/s on the 5090:
0.439 / (2910 x 3600) x 1e6 = $0.042 per million output tokens
# reserved (-35%) and cache-hit traffic push it lower stillProduction posture
- Baseline on reserved (−35%), burst on serverless — one balance, two curves.
- Health-check GET /health; restart-on-unhealthy is a template toggle.
- Weights on a read-only volume — replacement workers boot in seconds, and a bad deploy can never corrupt the library.
- Never serve on interruptible — that mode is for the training side of your pipeline.
Put the numbers to work
Every price in this guide is our live rate — fixed, ≥30% under the market median, billed per second. Deploy the exact setup above from the console in about 30 seconds, paid in crypto, no card and no KYC.
