Fine-Tune 8B Models on 4GB VRAM: Soup's Layer Streaming
The title '4GB VRAM Fine-Tuning an 8B Model' should be treated as marketing jargon by default—industry consensus is that QLoRA fine-tuning an 8B model requires at least 6.6–8GB VRAM (Unsloth's official best-case is 8GB, third-party tests on a 4090 with seq 2048 peak at 6.6GB). So we read through the Soup repository from README to benchmark files to core code, and the conclusion is: the claim holds, but there are many boundary conditions, and the author writes the costs more honestly than most projects.
Let's start with hard numbers. The author's tests on his development machine (RTX 3050 Laptop, 4GB VRAM, Windows 11): Llama-3.1-8B-Instruct, NF4 quantization, LoRA, batch 1, seq 512—peak VRAM 3.32GB, throughput 119.6 tok/s, GPU utilization 100%. The same configuration reproduced 113 tok/s and the same 3.32GB on an H100 (the H100 is slightly slower, indicating the bottleneck is not GPU compute at all). There's also a free Colab T4 notebook for reproduction that uses set_per_process_memory_fraction to hard-limit the process to 4.00GB, and the 8B model runs fine.
The Principle of Layer Streaming: Only One Layer in VRAM
Conventional fine-tuning (including QLoRA) requires the entire model weights to reside in VRAM, which is why the 8B model starts at 6.6GB. Soup's stream_layers: true takes a different approach:
- Frozen base weights reside in CPU memory (preferably page-locked/pinned; if memory is insufficient, it falls back to NVMe disks; SATA/mechanical disks are directly rejected), with only the LoRA adapter, its gradients, and optimizer states resident in VRAM
- Pre-allocate 2–8 VRAM buffers (default double buffering), perform asynchronous prefetching on a dedicated CUDA stream: when forward computing layer i, prefetch layer i+1; when backward computing layer i, prefetch layer i-1—overlapping weight transfer and computation, which is key to achieving 100% GPU utilization
- In the NF4 path, the base is quantized offline once and cached in shards; the cache key includes the quantization method and source checkpoint fingerprint to prevent streaming the wrong bytes
There's no free VRAM; the costs are hidden in two physical facts:
- Each layer must be read from memory twice per step (once for forward, once for backward recomputation—streaming forces gradient checkpointing). The code comment is straightforward: "dL/dx = Wᵀ·dL/dy, this is physics, not an implementation detail"
- Embeddings and lm_head are resident and not quantized—in the 8B peak of 3.32GB, this part accounts for 2.10GB. This explains why the 8B model is close to the 4GB ceiling and why the author hasn't tried 14B: only transformer layers can be saved, but vocabulary-related parts cannot
A detail shows engineering completeness: before training, there's a VRAM pre-flight predictor that, if over budget, refuses to run instead of letting you OOM (on Windows, exceeding VRAM doesn't error but silently slows down 9 times; this predictor is designed to prevent that). The prediction model is fitted from 10 real runs, with a worst-case error of 0.85%, biased only towards the safe side.
Getting Started: It Really Only Takes One YAML
Installation is via pip install soup-cli (currently v0.73.2). The minimal streaming configuration in the README:
training:
stream_layers: true # base streams in/out of VRAM; only the adapter trains
quantization: 4bit # NF4 — base ~4x smaller, so 8B fits in 4GB
batch_size: 4 # bigger batches amortise per-layer weight reads
stream_source: auto # RAM when it fits, NVMe when it does not
seed: 1234
Add base (model name), data (jsonl + alpaca format), lora (r/alpha) sections to run. Two practical conclusions directly from the author's benchmark: with the same effective batch, directly increasing batch_size is 2.52 times faster than gradient accumulation (1378 vs 540 tok/s, with VRAM increasing from 0.85GB to 2.28GB); training 1M tokens on an 8B model takes about 2.3 hours (the author notes this is an estimated value).
Boundary Conditions List (Determining If You Can Use It)
This part is more important than the principle, listed item by item:
- Speed cost is about 1.43 times (vs resident training, measured on 0.5B—the only size for fair comparison), with an additional ~9.8% overhead from per-layer NF4 dequantization. Preprint v3 has a respectable correction: the author originally thought the bottleneck was host-to-device transfer, but measurements disproved it (deleting all copies only sped up by 1.4%); the real streaming-exclusive cost is dequantization
- System memory is the new bottleneck: the 8B NF4 base requires page-locking about 3.6GB of memory; the author explicitly states 16GB system memory is the minimum
- Architecture whitelist: llama / qwen2 / qwen3 / mistral / gemma series / phi series; tasks only sft / dpo / orpo / simpo / kto; grpo and ppo are permanently unsupported—in RL's rollout generation phase, every token requires re-reading all layers, directly destroying the amortization premise of streaming
- Incompatible with Unsloth/mlx backends, also incompatible with DoRA, VeRA, packing. This means you're trading Unsloth's speed optimizations for VRAM
- The feature is still BETA, and it's essentially a solo project (745 out of 756 commits from the author alone)
- There's one timeliness flaw in the headline numbers (noted in the README itself): 119.6 tok/s was measured on v0.72.2; after v0.73.0 fixed a correctness bug (throughput -4.8% on 32B), it wasn't retested on 4GB cards
Why This Project Warrants a Separate Article: All Failure Records Made Public
Reading its benchmark directory is like reading a compilation of incident reports, which is very rare in open-source projects:
- Silent error in v0.72.0: the adapter key names from streaming training had an extra
.inner.segment, causing all loaders to silently load the unfine-tuned original model—PEFT only issued a UserWarning. Fixed in v0.72.1 - Gradient error in Issue #331: when using NF4 and a single layer exceeds about 165MiB (32B/72B scale), forward is bit-wise correct and loss curve normal, but gradients are secretly all wrong. The root cause is that bitsandbytes's
MatMul4Bitstores weights in ctx's regular attributes, bypassingsave_for_backward, causing aliasing with Soup's pooled buffers. Fixed in v0.73.0 - Verification that is bit-exact (bitwise identical) to resident training covers 9 architecture families × 2 precisions, with max absolute logit diff = 0.0—the above two bugs were caught by this type of verification
In the Show HN post (138 points), there were also two episodes: simonw questioned that the example directory's training data had only a few lines; the author responded that they were format samples and then discovered on the spot that 7 out of 8 example configurations used an old schema that couldn't be parsed at all, fixing it immediately and adding tests; the author was caught by the community using an LLM to write English replies early on (nearly half the comments were marked dead), admitted it, and switched to writing by hand—his native languages are Kazakh and Russian.
Most convincingly, the author's own discouragement: 'Don't buy a 4GB card for this. If buying a card, buy one with more VRAM; if the model can be resident in VRAM, don't enable streaming.' The real positioning of this feature is: you happen to have only a 4–6GB card and want to tinker with an 8B model locally.
Frequently Asked Questions (FAQ)
Is 4GB VRAM Fine-Tuning 8B Just a Gimmick?
No. The RTX 3050 Laptop 4GB tested peak at 3.32GB VRAM, 119.6 tok/s, with consistent results from H100 reproduction and a verifiable Colab notebook, and compared to the industry QLoRA minimum (6.6–8GB), it's indeed a breakthrough. But the conditions are fixed: NF4 quantization + LoRA + seq 512 + whitelisted architecture + 16GB system memory, with speed about 1.43 times slower than resident training.
What's the Relationship with Unsloth / QLoRA?
QLoRA solves 'weights are too large' (4-bit quantization), but the entire model still needs to be resident in VRAM; Unsloth builds on this for speed optimization, with a minimum of about 6.6–8GB for 8B. Soup's layer streaming solves 'resident' itself—only one layer + adapter in VRAM—so it can push the minimum down to 3.32GB, at the cost of being incompatible with Unsloth acceleration and slower. If you have 8GB VRAM, the Unsloth path remains the faster option.
Will Training Quality Be Compromised?
In terms of mechanism, no: layer streaming has been verified to be bit-exact (logit diff = 0) compared to resident training. The risk lies in engineering implementation—it has had two bugs where 'training looks normal but results are secretly wrong' (both fixed and publicly archived). When using it, it's recommended to do a small-sample inference comparison after training to confirm the adapter is effective.
When Should You Not Use It?
When running GRPO/PPO (permanently unsupported); when the model can be resident in VRAM (the author suggests not enabling streaming); using non-whitelisted architectures; system memory below 16GB; and in production pipelines sensitive to training speed.
