Running a 70B Model on a 4GB GPU: Hacker Dreams vs. Engineer Reality
Your GPU has 4GB of VRAM, and someone on the internet says it can run a 70B model. First instinct: clickbait. Mine too. Then I went through the AirLLM benchmarks and stopped being smug about it. Llama 3.1 405B on 8GB. DeepSeek-V3 (671B) on ~12GB. Kimi K3 — 2.8T parameters, the largest open-source model ever released — in 3.72GB of VRAM, measured end-to-end on an RTX 6000 Ada.
That same week Cloudflare published a post on serving Kimi K2.6 and GLM 5.2 at scale across their edge network. Quantize the KV cache, compress the weights, and suddenly one GPU holds twice the concurrent requests at roughly 30% less cost per token.
One camp is squeezing the worst GPU you own. The other is squeezing the most expensive one money can rent. Same argument underneath: actually using a model beats making it bigger.
How AirLLM "fakes" its way past the VRAM limit
No quantization. No distillation. No pruning. The trick is one sentence: load layers on demand, compute only what you need.
Classic inference stuffs the entire model into VRAM. A 70B model needs ~140GB just for weights, and a 4GB card dies on the spot. AirLLM flips it: slice the model into layers, load the layer you're computing from system memory, then throw it away.
MoE models get an even better deal. Kimi K3 has 2.8T parameters but only a handful of experts activate per token, so AirLLM streams just those. That's how it ends up at 3.72GB.
Think of it as borrowing one page at a time from the library instead of moving the whole building into your study. The books stay on the shelf; you check them out, read, and return.
You pay for this in speed. Every generated token means repeated disk and memory I/O, and throughput falls an order of magnitude or more below a fully-loaded deployment. Credit where it's due: AirLLM's own docs say so plainly — offline, low-concurrency, don't-hurry workloads.
Cloudflare's bet: make models live on the edge
AirLLM saves memory. Cloudflare saves compute and squeezes concurrency.
For a long-context MoE model like Kimi K2.6, the VRAM killer isn't the weights. It's the KV cache. Every token generated means storing attention keys and values for every token before it. Longer context, bigger cache, until the cache — not the model — blows up the GPU.
So Cloudflare dropped the cache from BF16 to FP8. Here's the part I like: per-request it's actually ~9% slower (137 vs 125 tok/s at single concurrency), which on its own looks like a bad trade. But BF16 runs out of memory at 32 concurrent requests, while FP8 keeps going to 64 and hits 2,192 tok/s — 41% higher than BF16's peak, at roughly 30% less cost per token.
Quality holds up too: GSM8K 94.24 → 94.09, MMLU 89.11 → 89.04. Noise-level differences.
Both roads are really about the same fight: cost
AirLLM serves the solo developer: debug locally, skip the A100 rental, get a demo running on whatever old card is already in the machine. Cloudflare lives in a different universe — thousands of requests across edge locations, where 30% off the bill is a digit, not a rounding error.
They're pulling in opposite directions and fighting the same fight: compute is never enough, so inference efficiency is the moat. Open source in 2026 doesn't lack models. It lacks cheap ways to run them. Whoever serves more requests per GPU — or runs bigger models on worse GPUs — wins.
Reality check: don't fall for the demo video
"4GB runs 70B" sounds heroic until you point it at production and latency humbles you. AirLLM trades speed for memory. It's for experiments and offline jobs, not serving traffic.
Cloudflare's playbook isn't portable either: FP8 KV cache quantization, disaggregated prefill/decode, SGLang tuning — that's distributed production engineering, not a weekend project.
And the part I care about most: both of these are engineering wins, not algorithmic breakthroughs. Engineering moves one notch at a time. It doesn't jump. So don't hold your breath for 70B on your phone tomorrow.
What you can do today
- Have an old GPU? Spend 20 minutes on an AirLLM demo (
pip install airllm). Judge it by real throughput, not the README. - Working on edge or serverless inference? Read the Cloudflare post carefully — the KV cache quantization and prefill/decode split pay for themselves in throughput.
- Neither applies? At least register the shift: inference is moving from "buy more GPUs" to "squeeze the ones you have." Next time you pick a model, price inference like a first-class citizen instead of finding out when the invoice lands.
Sources:
- AirLLM: 70B inference with single 4GB GPU
- Cloudflare: Smaller, faster, safer — running Kimi and GLM at scale
✨ 本文由 DeepSeek 生成初稿，Claude 审核润色。