---
title: "LLM VRAM requirements: how much GPU memory for 7B–405B models"
description: "Complete sizing tables for running and training LLMs — FP16, INT8 and 4-bit, with KV-cache math and the cheapest GPU that fits each model."
url: https://powergpu.ai/guides/llm-vram-requirements
last_modified: 2026-09-14T11:04:20+00:00
prices_as_of: 2026-09-14
site: PowerGPU (powergpu.ai)
---

Guide · Choosing hardware

# LLM VRAM requirements: how much GPU memory for 7B–405B models

Complete sizing tables for running and training LLMs — FP16, INT8 and 4-bit, with KV-cache math and the cheapest GPU that fits each model.

12 min read Published 2026-07-21 Updated 2026-09-03 (prices live from the sheet)

TL;DR

- Weights need 2 GB per billion parameters in FP16, 1 GB in INT8 and about 0.55 GB in 4-bit.
- Add 10-20 percent runtime overhead plus KV-cache: Llama 3.1 8B costs about 0.13 GB per 1k tokens per sequence.
- Practical fits: 8B is 18 GB FP16 or 6.5 GB in 4-bit; Llama 70B is 150 GB FP16, 44 GB in 4-bit.
- PowerGPU prices multi-GPU linearly, so two RTX 5090s at $0.439 each give 64 GB for 70B 4-bit serving.

## The 60-second rule of thumb

*the whole method*

```
weights_gb  = params_B x bytes_per_param      # fp16 2.0 · int8 1.0 · 4-bit ~0.55
runtime_gb  = weights_gb x 1.1..1.2           # CUDA ctx, activations, fragmentation
total_gb    = runtime_gb + kv_cache_gb        # see below — grows with context
```

That is genuinely the whole method. The table below applies it to the models people actually deploy, with the cheapest card on our sheet that fits each cell.

## The sizing table (inference)

| Model | FP16 | INT8 | 4-bit | Cheapest card that fits (4-bit) | $/hr |
| --- | --- | --- | --- | --- | --- |
| **Llama 3.2 3B** | 7 GB | 4.5 GB | 2.8 GB | [RTX 3060 · 12 GB](https://powergpu.ai/gpu/rtx-3060) | $0.042 |
| **Llama 3.1 8B** | 18 GB | 10 GB | 6.5 GB | [RTX 3090 · 24 GB](https://powergpu.ai/gpu/rtx-3090) | $0.108 |
| **Qwen 2.5 14B** | 31 GB | 17 GB | 10.5 GB | [RTX 4090 · 24 GB](https://powergpu.ai/gpu/rtx-4090) | $0.327 |
| **Qwen 2.5 32B** | 70 GB | 37 GB | 21 GB | [RTX 5090 · 32 GB](https://powergpu.ai/gpu/rtx-5090) | $0.439 |
| **Llama 3.1 70B** | 150 GB | 78 GB | 44 GB | [H100 PCIE · 80 GB](https://powergpu.ai/gpu/h100-pcie) | $1.867 |
| **Qwen 2.5 72B** | 155 GB | 80 GB | 46 GB | [H100 PCIE · 80 GB](https://powergpu.ai/gpu/h100-pcie) | $1.867 |
| **Mistral Large 123B** | 260 GB | 133 GB | 78 GB | [H200 · 141 GB](https://powergpu.ai/gpu/h200) | $2.791 |
| **Llama 3.1 405B** | 850 GB | 440 GB | 245 GB | [B200 · 192 GB](https://powergpu.ai/gpu/b200) | $5.425 |

Weights + 15% overhead, excluding KV-cache; 405B rows assume multi-GPU sharding (per-card price shown). Figures are engineering estimates for planning, not benchmarks.

## KV-cache: the part everyone forgets

Serving crashes rarely come from weights — they come from context. Every token in flight stores keys and values for every layer:

*why the 24 GB card OOMs at batch 32*

```
kv_gb = 2 x layers x kv_heads x head_dim x bytes x context x batch / 1e9

# Llama 3.1 8B (GQA, fp16): ~0.13 GB per 1k tokens per sequence
# 32 concurrent chats x 8k context ≈ 33 GB of cache — MORE than the weights
```

Levers, in order of cheapness: quantize the cache (FP8 KV halves it), cap concurrent context (vLLM's --max-num-batched-tokens), then buy VRAM. This is why serving pages recommend 32–48 GB cards for "models that fit in 16 GB".

## Training & fine-tuning VRAM

| Method | VRAM ≈ | 8B lands on | 70B lands on |
| --- | --- | --- | --- |
| Full fine-tune, AdamW fp16 | 16 GB / B params | 8× 24 GB or 2× 80 GB | 16× 80 GB (cluster) |
| Full, 8-bit optimizer | ~10 GB / B | 1× 80 GB | 9× 80 GB |
| LoRA (fp16 base) | weights + ~2 GB | 1× 24 GB | 2× 80 GB |
| QLoRA (4-bit base) | ~0.7 GB / B + 2 GB | 1× 12 GB | 1× 48–80 GB |

Gradient checkpointing trades ~20% speed for ~30% memory and is on by default in axolotl configs — the [QLoRA walkthrough](https://powergpu.ai/guides/fine-tune-llm-qlora) shows real consumption at each step.

## When one card is not enough

- **Tensor parallelism** (vLLM --tensor-parallel-size) splits layers across 2–8 GPUs on one machine — NVLink helps but PCIe 4/5 serves fine to 4×.
- **Our pricing is linear**: 2× [RTX 5090](https://powergpu.ai/gpu/rtx-5090) (64 GB total) costs exactly 2× $0.439 — often the cheapest 70B-4bit rig on the sheet.
- **Past 8 GPUs**, you are in [cluster](https://powergpu.ai/products/clusters) territory: pipeline or FSDP sharding over InfiniBand.

Cross-check any plan against the per-card fit lists on the [GPU pages](https://powergpu.ai/gpus) — each fiche computes what it can hold from these same formulas.

---

Put the numbers to work

Every price in this guide is our live rate — fixed, ≥30% under the market median, billed per second. Deploy the exact setup above from the [console](https://cloud.powergpu.ai/) in about 30 seconds, paid in crypto, no card and no KYC.

---

*About PowerGPU:* PowerGPU (powergpu.ai) is a cloud GPU rental service offering 80 NVIDIA GPU models — from the RTX A2000 at $0.024/hr to the B300 — at fixed prices set at least 30% below the public GPU marketplace median and re-checked weekly (H100 SXM: $1.428/hr on-demand). Billing is per second with no minimums; payment is crypto only (USDT, BTC, XMR, ETH, SOL, LTC, TRX) with no KYC. Instances run in Tier-III datacenters across 32 regions with a 99.9% uptime SLA and deploy in about 30 seconds from the web console (cloud.powergpu.ai) or the REST API.

Source: https://powergpu.ai/guides/llm-vram-requirements · Site index for AI assistants: https://powergpu.ai/llms.txt · Full content: https://powergpu.ai/llms-full.txt