How Large a Local LLM Can a 24GB Mac mini Run? A Summary of Memory Budgets, Measured Speeds, and Acceleration Methods
How Large a Local LLM Can a 24GB Mac mini Run? A Summary of Memory Budgets, Measured Speeds, and Acceleration Methods
Direct answer: The ceiling for 24GB unified memory is 4-bit quantization of 27B-scale models—not theoretical speculation. We successfully ran Qwen3.8-27B (4-bit, about 15GB model size) along with a 2B draft model on a base M4 Mac mini, with peak memory 19.4GB, provided large-memory applications like browsers are closed. In terms of speed, pure autoregressive decoding is 6.5 tok/s ('barely watchable'), and with speculative decoding, it reaches 11.7-12.2 tok/s, close to normal reading speed.
This article is the summary page for our Mac mini local deployment series: how to calculate memory budgets, what to expect for each model size, and which of three acceleration methods actually works. Each conclusion links to the corresponding full measurement.
1. Memory Budget: How Much Memory Is Actually Available for Models
Unified memory is shared between CPU/GPU, and the system itself and resident applications take a chunk. From our testing experience: it's feasible to allocate 19-20GB for models and inference stacks, but you need to actively close large applications; in a daily state with resident Chrome + IDE, the safe budget is around 16GB.
The estimation formula for model file size (including format overhead, see the quantization guide for details):
Size ≈ parameter count × bytes per parameter (Q4_K_M about 0.6 bytes, Q6_K about 0.8, Q8_0 about 1.06)
On top of that, add KV cache (grows linearly with context length) and runtime overhead. Based on this calculation, here's how each model size fares on 24GB:
| Model Size | Recommended Quantization | Model Size on Disk | Experience on 24GB |
|---|---|---|---|
| 7-9B | Q8_0 / Q6_K | 6-10GB | Comfort zone: high quantization + long context + no need to close apps |
| 13-14B | Q6_K / Q5_K_M | 8-11GB | 从容 (Carefree): still have margin for long context |
| 20B-scale | Q5 / Q4 | 11-14GB | Feasible: moderate control over context length |
| 27-32B | Q4_K_M / 4-bit | 15-18GB | Ceiling: can run, but need to close large apps and control context (27B measured peak 19.4GB) |
| 70B+ | — | Q4 needs 40GB+ | Cannot run, don't try |
A counterintuitive reminder: when memory is insufficient, downgrading quantization is worse than switching to a smaller model. Compressing 27B to Q2 severely distorts the probability distribution, usually inferior to 14B Q5—see the quantization accuracy loss guide for supporting data.
2. Speed Expectations: The Bottleneck Is Memory Bandwidth, Not Compute
Local LLM decoding is a memory bandwidth-bound task: every token generated requires moving all activated weights from memory. The base M4 has 120GB/s bandwidth, M4 Pro is 273GB/s—the same model will run noticeably faster on the Pro version; but acceleration ratio conclusions (next section) are determined by the algorithm and are cross-device general.
Fixed numbers for base M4 + Qwen3.8-27B 4-bit (three rounds of retesting, within 5% fluctuation):
- Pure autoregressive: 6.5 tok/s—reading requires waiting for characters to appear
- Speculative decoding (DFlash 2 draft model): 11.7-12.2 tok/s, stable 1.8-1.9x—close to normal reading speed
Halving parameter count roughly doubles speed (a direct inference from bandwidth bottleneck); 7-9B models on this machine can achieve smooth conversation speeds without pressure.
3. Real Effects of Three Acceleration Methods (All Measured on the Same Machine)
Route 1: DFlash 2 Speculative Decoding—Recommended, 1.8-1.9x ✅
Use a 2B draft model to propose candidates for the 27B main model, with main model batch verification—mathematically lossless. Measured speed increase from 6.5 to 11.7-12.2 tok/s. Note two pitfalls: quantized model block-size should not exceed 5 (official cliff warning confirmed in testing: block-size 8 drops to 1.11x); use 4-bit for draft model, not 8-bit (draft only proposes candidates; high precision doesn't improve acceptance rate and instead slows output).
👉 Full deployment commands and three rounds of data: 24G Mac mini Running Qwen3.8-27B + DFlash 2: Measured 1.8x
Route 2: Qwen3.8 Native MTP—Saves Memory but Slower ⚠️
Qwen3.8's built-in MTP (Multi-Token Prediction) head theoretically eliminates the draft model. Head-to-head testing on the same machine: saved 3GB of memory, but speed dropped 24%—currently not cost-effective under MLX implementation, unless you're at the memory threshold and prefer slower over crashing.
👉 Comparison data: Qwen3.8 Native MTP vs DFlash 2 Head-to-Head on Same Machine
Route 3: Switching to llama.cpp for Speculative Decoding—This Doesn't Work ❌
Same machine, same DFlash 2 draft: MLX accelerated 1.9x, but on llama.cpp it slowed down 50% and even OOM'd. To benefit from speculative decoding on Apple Silicon, stay within the MLX ecosystem for now.
👉 Reproduction process and analysis: Same DFlash 2: MLX Accelerates, llama.cpp Decelerates
4. Frequently Asked Questions (FAQ)
Is it worth running 27B on the base M4 Mac mini (24GB)?
Yes, but recognize its position: 11-12 tok/s is 'fluent reading' not 'instant response', and you need to close large apps. If budget allows, the same capacity M4 Pro's 273GB/s bandwidth can boost absolute speed by more than double; if you just want a fluent daily local assistant, 7-14B models offer a more relaxed experience on the base model.
Can clustering several base Mac minis run larger models?
Technically, there are such solutions (distributed inference projects like exo), but we have not tested them and provide no numbers. In principle, a reality check: decoding is bandwidth-bound, and cross-machine communication (even via Thunderbolt) is much slower than unified memory; clusters typically solve 'fitting in' not 'running fast'. A single 24GB machine can already fit 27B 4-bit; before clustering for larger models, calculate communication overhead first.
What can a 16GB Mac mini run?
Using the formula from Section 1 minus system overhead, the comfort zone is 7-9B (Q5/Q6), and 14B Q4 is the ceiling. Don't even think about 27B.
Do I need to install CUDA or graphics drivers?
No. Apple Silicon uses Metal, and both MLX and llama.cpp natively support it; just run pip install mlx-lm or brew install llama.cpp to start.
How long should I set the context length?
KV cache consumes memory linearly with context length. On 24GB, measured for 27B is 'control at thousands of tokens, peak 19.4GB'; for 7-9B models, you can expand to tens of thousands of tokens. Specific numbers vary greatly with model architecture; refer to peak memory printed by the inference framework.
Series Article Index
- 24G Mac mini Running Qwen3.8-27B + DFlash 2: Measured 1.8x, and Why Not Official 3x
- 24G Mac mini Measuring Qwen3.8 Native MTP: Saved 3GB Memory but Speed Dropped 24%
- Same 24G Mac mini, Same DFlash 2: MLX Accelerates 1.9x, llama.cpp Decelerates 50% and OOMs
- How Much Accuracy Is Lost in LLM Quantization? One Table Explains Q8 to Q2
- Qwen3.8-27B Overthinking Issue and reasoning_effort Measurement