Qwen3.8 Native MTP on a 24GB Mac mini: 3GB Less Memory, 24% Slower — Head-to-Head With DFlash 2

Qwen3.8 Native MTP on a 24GB Mac mini: 3GB Less Memory, 24% Slower — Head-to-Head With DFlash 2

Background

My DFlash 2 write-up got speculative decoding running on this 24GB Mac mini M4 at 1.8–1.9x (6.5 → 11.7–12.2 tok/s), and ended with a promised follow-up:

Qwen3.8's native MTP head: the model ships with a trained multi-token-prediction head, llama.cpp already supports mounting it, no extra 2B draft model needed — the more memory-frugal option on a 24GB machine. The trade-off is a different speedup profile, worth a separate head-to-head.

This is that head-to-head, and the headline is the opposite of what I expected: on this machine, native MTP is not "a bit less speedup for less memory" — it is a net slowdown. Prose generation drops from 6.0 to 4.5 tok/s (-24%), and code generation doesn't pay either (-4%). The memory saving is real: 16.0GB peak versus 19.4GB on the DFlash route, 3.4GB less.

Same machine, same prompt, same greedy-400-token method. Two speculative decoding routes: one 1.9x, the other 0.76x. The gap has a specific, measurable cause.

The analysis

Both routes share the same principle: guess a few tokens cheaply, then have the big model verify the whole block in one forward pass. The difference is who guesses:

• DFlash 2: an external ~2B-parameter draft model (a 3.85GB BF16 download, quantized to 4-bit at load, ~1GB resident), running on the MLX backend.
• Native MTP: Qwen3.8 was trained with multi-token-prediction layers (blk.*.nextn.* tensors), and quantized GGUFs keep them. llama.cpp added draft-mtp speculative decoding in PR #22673 (July 2026): without the flag those tensors load and sit idle; with it, they become the draft head.

Speculative decoding only pays under one approximation: verifying n+1 tokens costs about the same as verifying 1 — the weights are read once and shared across rows. How well that holds varies wildly by backend, and it turns out to be the whole story here.

Before touching anything I checked community data. sudoingX/qwen38-mtp collects 53 A/B configs: on CUDA/ROCm cards the flag is worth +33% to +145%. The single Apple M4 24GB (Metal) row reads 5.8 → 5.8, a wash, and the accompanying Apple Silicon deep-dive shows the split underneath: code +9–10%, prose -22–24%, cancelling out. That repo lists "MLX versus llama.cpp on the same Mac" as its top open thread — and my machine, having already produced the DFlash numbers, is positioned to answer it.

Approach and choices

Why llama.cpp rather than MLX: first-party support for the native MTP head lives in llama.cpp (PR #22673, merged to master); mlx_lm and the dflash CLI currently have no ready path to mount Qwen3.8's nextn head. So this is really a whole-stack comparison — "llama.cpp + native MTP" versus "MLX + DFlash draft model" — not a single-variable experiment. Each stack gets its own autoregressive baseline, and each speedup ratio is computed against its own baseline.

GGUF choice: two sources work — ggml-org/Qwen3.8-27B-GGUF ships separate mtp-*.gguf draft files (Q4_0, 1.68GB, mounted via --spec-draft-model), while unsloth/Qwen3.8-27B-GGUF embeds the nextn tensors in the main GGUF, one file total. I picked unsloth's UD-Q4_K_S (15.36GB): embedded is simpler, and the size is the closest match to the MLX 4-bit weights (~15GB) from the last article, which keeps the memory comparison honest. ggml-org's smallest main quant is Q4_K_M at 18.97GB — tight on a 24GB machine, skipped.

Ruled out: Q8_0 (29GB) and BF16 (54GB) don't fit. The community already swept deeper draft depths on this exact chip and found them worse; I only test n-max 2 and 4 to confirm the direction.

Test procedure

Environment: Mac mini M4 base model, 24GB, macOS 26.3.1. llama.cpp built from that day's master (b96806d):

brew install cmake
cd ~/llm && git clone --depth 1 https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_METAL=ON -DLLAMA_CURL=OFF
cmake --build build --config Release -j 8 -t llama-server llama-batched-bench

Model download (15.36GB):

curl -L -C - -o Qwen3.8-27B-UD-Q4_K_S.gguf \
  "https://huggingface.co/unsloth/Qwen3.8-27B-GGUF/resolve/main/Qwen3.8-27B-UD-Q4_K_S.gguf"

The two server configurations differ by exactly two flags:

# baseline (autoregressive decoding)
llama-server -m Qwen3.8-27B-UD-Q4_K_S.gguf -c 8192 -ngl 999 -fa on \
  -b 512 -ub 512 --parallel 1

# native MTP speculative decoding
llama-server -m Qwen3.8-27B-UD-Q4_K_S.gguf -c 8192 -ngl 999 -fa on \
  -b 512 -ub 512 --parallel 1 --spec-type draft-mtp --spec-draft-n-max 2

Three flags matter: -b 512 -ub 512 is a community lesson paid for in OOMs on this exact machine (the default -b 2048 blows the Metal command buffer mid-generation and takes the server down); --parallel 1 is measurement discipline (speculative decoding is a single-stream optimization; a concurrent baseline reads low and inflates your gain). The startup log confirms the MTP head is mounted:

Measurement follows the same "net speed" protocol as the DFlash article: the same prose prompt (a ~300-word English essay task) plus a new code prompt (a Python LRU cache implementation), greedy (temperature 0), exactly 400 generated tokens, two rounds per arm. Speed is timings.predicted_per_second from llama-server — generation phase only, equivalent to the total-minus-load-calibration method. Thinking mode is disabled via chat_template_kwargs: {"enable_thinking": false}. Peak memory comes from /usr/bin/time -l.

Results

One real terminal capture covers all three arms:

Side by side with the published DFlash 2 numbers from the same machine:

Four points worth unpacking:

1. The drafts are fine; the verify step loses the money. Acceptance of 59–85% with mean accepted lengths of 2.2–3.9 sits right in the range the community's matched CUDA control reports (0.46–0.93) — and the same quality of drafts buys +69–82% on an RTX 3090. Where the money goes is directly measurable with llama-batched-bench:

Batch-1 decode runs 6.10 tok/s; batch-8 aggregate throughput is only 6.90 tok/s — 1.13x amortization where the ideal is 8x. Verifying 8 rows on Metal costs roughly 7 full forward passes. The core premise of speculative decoding — block verification is nearly free — simply does not hold on this backend: verifying 3 rows (n-max 2 means 2+1) costs nearly 3x, so no acceptance rate can claw the draft cost back. The community deep-dive traces the lead to a Metal kernel gate (ne11_mm_min = 8: batches under 8 rows never reach the matrix-matrix kernel, and K-quants face an extra restriction on the small-batch path) — which is also why the same quantized model amortizes at 3.34x for batch 8 on CUDA.

2. Prose loses much more than code — same direction as the community, different magnitude. Prose has low acceptance (59%) and many verify rounds; my -24% matches the community's -22–24% exactly. On code I measured -4% where they measured +9–10%. The direction ("code far better than prose") agrees; the sign difference most likely comes down to the prompt (my LRU cache task accepted 2.70 tokens per draft on average; their code tasks ran longer) — which itself says that even the best case on Metal hovers around break-even.

3. A counterintuitive footnote on greedy losslessness. At n-max 2, the 400-token output is byte-identical to baseline for both prompts. At n-max 4, the prose output diverges at character 1294 ("More critically," becomes "Furthermore,") — and that divergence itself reproduces exactly across rounds. Speculative decoding is lossless in distribution; different verify widths take different Metal kernel paths with slightly different floating-point results, and greedy sampling can flip wherever two tokens are nearly tied. Quality is unaffected, but byte-for-byte reproducibility only holds within a fixed configuration.

4. The memory ledger is as promised. What you save is exactly the 2B draft model: 16.0GB vs 19.4GB. Note that the llama.cpp baseline itself runs ~7% below MLX (5.97 vs 6.45 tok/s) — two inference stacks at different maturity on this hybrid-attention architecture; the community notes these Metal kernels are "young and the floor should rise with upstream work."

One-line verdict: if you want speculative-decoding gains on a 24GB Mac today, use DFlash 2 + MLX. The native MTP head on Metal is "healthy acceptance, unprofitable verification" — 3GB of memory saved at the price of a quarter of your prose throughput. Unless your workload is pure code generation and you've measured your own break-even, leave the flag off.

Pitfalls

In the order I hit them — three scripting traps, two experiment-design traps:

1. zsh's echo eats \n inside JSON. Storing a curl response in a variable and writing it with echo "$resp" > file lets zsh's builtin echo interpret backslash escapes, turning \n inside JSON strings into literal newlines — json.load fails with "Invalid control character". Write with curl -o file directly (or printf '%s').
2. Under /usr/bin/time -l, kill -INT $! signals the time wrapper, not llama-server. time doesn't forward SIGINT; the server stays alive and the script's wait hangs forever. Fix: pkill -INT -f "llama-server.*--port $PORT" targets the real process by name — peak RSS still gets reported.
3. During model load, llama-server's /health returns 503 — but curl's exit code is 0. Probing with curl -s -o /dev/null && break passes while the model is still loading, and every subsequent request gets a 503 error JSON with zeroed timings. You must check the body: curl -s .../health | grep -q '"ok"'.
4. The script command's output file directory must exist before it starts — otherwise the whole capture session fails immediately. A mkdir -p inside the captured script is too late; the capture never begins.
5. The default batch size blows up, and load time won't warn you. Community-documented on this exact chip: at -b 2048, load-time allocation is a comfortable 13.9GB, then a transient prompt-batch allocation mid-generation overflows Metal (kIOGPUCommandBufferCallbackErrorOutOfMemory) and the server dies on an assert. I adopted -b 512 -ub 512 upfront and saw zero OOMs.

Still worth watching

• Native MTP on MLX: a community challenge has contributors reporting 77 tok/s with MTP on Apple hardware (machine unspecified, likely Ultra-class), implying MLX's verify path amortizes. If that holds, "native MTP doesn't work on Macs" needs rewriting as "llama.cpp's Metal kernels don't, yet" — exactly the open question the community deep-dive flags as its number one.
• llama.cpp's draft-dflash: while reading the --spec-type option list I noticed DFlash support has been merged into llama.cpp (it was still a PR when the last article went out). GGUF-stack DFlash versus MLX-stack DFlash is another same-machine comparison waiting to happen.

All performance figures are first-hand measurements taken 2026-09-02 on a Mac mini M4 base 24GB (macOS 26.3.1, llama.cpp b96806d built from that day's master), using the same protocol as the August DFlash 2 test (greedy, 400 tokens, generation-phase net speed); community comparison data is cited to its original sources. Metal support for this architecture is evolving quickly — these conclusions have a shelf life.