TL;DR
- An RTX 3060 12 GB is the cheapest card that runs Ollama well, covering every 7B-8B model and most 14B models at Q4.
- Ollama pulls 4-bit weights by default, so budget about 0.6 GB per billion parameters plus 1-3 GB for context.
- That puts 8B on 12 GB cards, 27B-32B on 24 GB, and 70B on a 48 GB RTX A6000 at $0.281 per hour on PowerGPU.
- Ollama is single-stream: expect about 120 tokens per second for 8B on an RTX 4090, and use vLLM when many users share the card.
How much VRAM each Ollama model needs
Ollama pulls 4-bit (Q4_K_M) weights by default, so the rule of thumb is about 0.6 GB per billion parameters, plus 1–3 GB for context. That puts 8B models on 12 GB cards, 14B on 12–16 GB, 27B–32B on 24 GB, and 70B on 48 GB. Go up a class if you want 32k+ context or FP16 weights (:fp16 tags need 2 GB per billion).
The table: model → cheapest card → speed → cost
Tokens per second are indicative single-stream numbers for Ollama with the whole model on the GPU (llama.cpp CUDA backend); your prompt length and quantization move them. Prices are today's on-demand rates; interruptible halves them.
| Ollama model | Weights | Card | On-demand | ~tok/s | $ / M tokens | Note |
|---|---|---|---|---|---|---|
| llama3.1:8b | 4.9 GB | RTX 3060 12 GB | $0.042 | ~35 | $0.33 | The cheapest card that runs 8B comfortably |
| llama3.1:8b | 4.9 GB | RTX 4090 24 GB | $0.327 | ~120 | $0.76 | Snappy chat; room for 32k context |
| qwen2.5:14b | 9.0 GB | RTX 4070 12 GB | $0.075 | ~45 | $0.46 | 12 GB is enough at Q4 |
| qwen2.5:14b | 9.0 GB | RTX 5090 32 GB | $0.439 | ~110 | $1.11 | 1.79 TB/s GDDR7 shows in tokens/s |
| gemma3:27b | 17 GB | RTX 3090 24 GB | $0.108 | ~25 | $1.20 | 24 GB with context headroom at the lowest rate |
| qwen2.5:32b | 20 GB | RTX 4090 24 GB | $0.327 | ~40 | $2.27 | Tight but fine on 24 GB with 8k context |
| qwen2.5:32b | 20 GB | RTX 5090 32 GB | $0.439 | ~62 | $1.97 | 32 GB gives long-context breathing room |
| llama3.3:70b | 43 GB | RTX A6000 48 GB | $0.281 | ~15 | $5.20 | Cheapest single card that holds 70B |
| llama3.3:70b | 43 GB | L40S 48 GB | $0.514 | ~20 | $7.14 | Datacenter card for an always-on assistant |
| llama3.3:70b | 43 GB | H100 PCIE 80 GB | $1.867 | ~38 | $13.65 | HBM bandwidth — the fastest single-card 70B |
Single-stream $/M tokens looks expensive next to API providers because one chat at a time leaves most of the card idle — Ollama is a private-assistant tool, not a serving engine. For many users, vLLM batches requests and drops the cost per token by 10–30×.
An always-on private assistant, per month
| Assistant | Card | 24/7 on-demand | Reserved (3 mo) | 8 h/day, workdays |
|---|---|---|---|---|
| 8B chat (llama3.1:8b) | RTX 3060 | $31 | $20 | $7 |
| 14B–32B (qwen2.5:32b) | RTX 4090 | $239 | $155 | $58 |
| 70B (llama3.3:70b) | RTX A6000 | $205 | $133 | $49 |
| 70B, fast (llama3.3:70b) | H100 PCIE | $1,363 | $885 | $329 |
Add a volume for the model library (/root/.ollama, 30–100 GB at $0.08/GB/month) so pulled models survive stop/start and instance swaps.
Setup in one command
powergpu launch --gpu rtx-3060 --template ollama --disk 40 --volume ollama:/root/.ollama
# ✓ instance i-2b7e11c0 running (26.4s) · $0.042/hr
# https://i-2b7e11c0.powergpu.ai:11434 (Ollama API, TLS)
curl https://i-2b7e11c0.powergpu.ai:11434/api/pull -d '{"name":"llama3.1:8b"}'Want a chat UI on top? The Open WebUI template bundles Ollama with a multi-user interface, RAG and model management behind TLS.
Five ways to spend less
- Match the card to the model, not the ambition. An 8B assistant on an H100 is 40× the price of the same assistant on a 3060 and does not answer better.
- Stop it when nobody is talking. Per-second billing means a 9-to-5 assistant costs a third of a 24/7 one; the disk keeps the models.
- Reserve if it never sleeps. −35% from three months — $0.027/hr for the 3060, $0.182/hr for the A6000.
- Q4 over FP16. Half the VRAM class, nearly the same answers for chat; keep FP16 for evaluation runs.
- Two cheap cards can beat one expensive one. 2× RTX 4090 (48 GB) at $0.654/hr runs 70B for less than one L40S — slower per token, cheaper per month.
Put the numbers to work
Every price in this guide is our live rate — fixed, ≥30% under the market median, billed per second. Deploy the exact setup above from the console in about 30 seconds, paid in crypto, no card and no KYC.
