How Much VRAM Do I Need for a Local LLM? (2026 Guide with Calculator)
How Much VRAM Do I Need for a Local LLM? (2026 Guide with Calculator)
Every open-weight model release kicks off the same scramble: people open a Reddit thread, check the model card, and ask "will this run on my GPU?" The honest answer is almost never a clean yes or no — it depends on four things you have to multiply together: parameter count, quantization, context length, and runtime overhead. This guide walks through each one, gives you the rule of thumb, and points you at a calculator that does the exact math.
What Actually Determines VRAM Requirements for a Local LLM
VRAM requirements for local LLM inference come from four moving parts:
- Model weights — the parameters, stored at a precision determined by your quantization choice
- KV cache — keys and values for every token in the context, stored for every layer
- Runtime overhead — buffers, activations, and memory fragmentation (roughly 8% of weights with a 1.5 GB floor)
- Batch size — this guide assumes batch size 1 (single-user inference), which is what most local setups use
Too many calculators only count weights. At long context lengths, the KV cache can match or exceed the weights — skipping it badly underestimates what you actually need.
The Rule of Thumb: 0.6 GB per Billion Parameters at Q4_K_M
Quantization matters more than the model name. At Q4_K_M (~4.8 bits per weight), weights take roughly 0.6 GB per billion parameters:
- 7–8B model → ~5 GB → fits on an 8 GB card
- 13B model → ~8 GB → 12 GB card is comfortable
- 30B-class model → ~18–20 GB → needs a 24 GB card
- 70B model → ~42 GB → two 24 GB cards, or a 64 GB+ Mac
- 671B model (DeepSeek R1 full) → ~380 GB → 5–6× H100 80GB
These numbers are weights only. Add the KV cache for your context length and overhead before trusting them.
Quantization: The Cheapest Way to Cut VRAM
Quantization shrinks the bytes per weight. FP16 stores 2 bytes per parameter; everything else is a trade of quality for memory:
| Format | Bytes per param | vs FP16 | Typical use |
|---|---|---|---|
| FP16 | 2.0 | 1× | Reference, maximum quality |
| Q8_0 | 1.0 | 2× smaller | Near-lossless GGUF |
| Q6_K | ~0.8 | 2.5× smaller | High-quality GGUF |
| Q5_K_M | ~0.66 | 3× smaller | Good quality / size balance |
| Q4_K_M | ~0.6 | 3.3× smaller | Community default |
| MXFP4 | ~0.53 | 3.8× smaller | Native format for Kimi K3, gpt-oss |
A model that needs 140 GB of weights at FP16 drops to around 42 GB at Q4_K_M. That is the difference between renting a multi-GPU server and running it on a single 48 GB card.
Q4_K_M is the community default for good reason: modest quality loss, huge memory savings. Go Q5_K_M or Q6_K if you have headroom, Q8_0 for near-lossless, and only drop to Q3/Q2 when nothing else fits — quality degrades noticeably below 4 bits.
Context Length: The Part Everyone Forgets
The KV cache grows linearly with context length. For a dense 70B model at FP16, 128K tokens of context needs roughly 40 GB of KV cache — enough to double your total VRAM requirement on its own.
Two architectural notes:
- MLA (Multi-head Latent Attention) models — DeepSeek and Kimi compress the KV cache dramatically. This is why a trillion-parameter MoE can have a smaller context footprint than a dense 70B.
- FP8 KV cache halves the cache again on hardware that supports it (RTX 4090+).
MoE vs Dense: Parameter Count Isn't the Whole Story
Mixture-of-Experts models (Kimi K3, DeepSeek R1, Qwen3-MoE, gpt-oss) have enormous total parameter counts but activate only a fraction per token. That does not mean you can store fewer weights — the full parameter set still has to live in VRAM, because any expert can be routed to. What MoE does change is the KV cache and compute behavior, which is why an LLM VRAM calculator that treats every model as dense will mislead you on the new flagships.
Real-World Examples
Can I run DeepSeek R1 on a single GPU? Not the full 671B model. At Q4 quantization the weights alone are about 380 GB, needing roughly 5–6× H100 80GB. Single-GPU users should run distilled variants — R1-Distill-Qwen-32B fits on a 24 GB card at Q4_K_M.
Can I run Kimi K3 locally? No consumer hardware can. K3 is a 2.8-trillion-parameter MoE shipped in MXFP4 — weights alone are ~1.4 TB, so even at native quantization you need a multi-GPU server (around 8× H200 141GB depending on context). The practical way to use K3 is a hosted API.
Can I run a 7B model on an 8 GB card? Yes, at Q4_K_M with a moderate context length. Weights are ~5 GB, leaving ~3 GB for KV cache and overhead — roughly 8K–32K context depending on the model's KV compression.
Use the Calculator Instead of Guessing
The rule of thumb gets you in the ballpark. When the decision is real — renting a GPU, buying a Mac for local inference, starting a 1 TB weight download — use the LLM VRAM Calculator to get an exact estimate: pick the model preset or enter parameters, choose your quantization and context length, and it returns weights + KV cache + overhead, plus a "will it fit" table across RTX 4090/5090, A100/H100/H200/B200, and Apple Silicon Macs.
The estimates are typically within ~10% of real usage at batch size 1. Actual numbers vary by runtime (llama.cpp offloads layers, vLLM pre-allocates KV blocks, MLX on Apple Silicon shares unified memory with the OS), so treat the output as a planning figure, not a guarantee.
FAQ
How much VRAM do I need for a local LLM? At Q4_K_M, roughly 0.6 GB per billion parameters for weights alone. An 8B model needs about 5 GB, a 70B model about 42 GB — before KV cache and overhead.
How much VRAM does quantization save? FP16 stores 2 bytes per parameter. Q8_0 halves it, Q4_K_M cuts it to ~0.6 bytes (3.3× smaller), and MXFP4 averages ~0.53 bytes. A 140 GB FP16 model becomes ~42 GB at Q4_K_M.
Does the estimate include the KV cache and overhead? Yes, in a full calculator. Total = weights + KV cache at your context length + runtime overhead (~8% of weights, 1.5 GB floor). Weight-only calculators badly underestimate long-context usage.
Which GGUF quantization should I choose? Q4_K_M is the community default. Q5_K_M / Q6_K if you have spare memory, Q8_0 for near-lossless, Q3/Q2 only when nothing else fits.
How does context length affect VRAM? The KV cache grows linearly with context. A dense 70B at FP16 needs ~40 GB of cache at 128K tokens — potentially doubling your total requirement. MLA models (DeepSeek, Kimi) compress this dramatically, and FP8 KV cache halves it again.
Is my data uploaded anywhere? No. The calculator runs entirely in your browser — model specs are bundled with the page and all math happens locally.