Same 24GB Mac mini, Same DFlash 2: 1.9x Faster on MLX, 50% Slower (and OOM) on llama.cpp

Background

This is the third speculative-decoding benchmark on the same base Mac mini M4 with 24GB. Previously:

• DFlash 2 on MLX: external 2B draft model, 6.5 → 11.7–12.2 tok/s, a stable 1.8–1.9x;
• Qwen3.8 native MTP on llama.cpp: a net slowdown (prose -24%), root-caused with llama-batched-bench to Metal's batched-decode amortization being only 1.13x.

The MTP article left an obvious open question. llama.cpp merged DFlash 2 support in PR #27342 (--spec-type draft-dflash), and z-lab published official GGUF draft models for it. Same DFlash 2 algorithm, but on the llama.cpp stack — does it replicate MLX's 1.9x, or repeat MTP's net loss?

The experiment discriminates between two hypotheses, and only one can survive. If MTP lost because of draft quality, then DFlash 2 — a much stronger drafter (official CUDA eval: acceptance length 5.1–5.4) — should win. If MTP lost to the backend tax of unamortized batch verification on Metal, no drafter can help.

Spoiler: the backend tax won, and by more than expected. The README-recommended --spec-draft-n-max 7 hits a reproducible Metal OOM on this machine. The only configuration that runs stably, n-max 3, drops prose from 6.0 to 3.0 tok/s (-50%). On the code prompt the acceptance rate reaches 83.8% — 3.5 tokens landed per step — and it still loses 23%. On the same machine, MLX delivers 1.8–1.9x.

Analysis

Three articles in, the speculative-decoding landscape on this machine is a 2×2 grid:

Part two already quantified llama.cpp Metal's core problem: batch verification doesn't amortize. llama-batched-bench measured batch-8 aggregate throughput at 6.90 tok/s versus 6.10 single-stream — an amortization ratio of just 1.13x (the same scenario on CUDA gives 3.34x). Speculative decoding's profit formula assumes "verifying n draft tokens costs about the same as verifying one." On this backend, that assumption is false.

So the pre-test prediction was: DFlash 2's draft quality must still pass through the verification toll booth. The measured bill turned out to have a second line item — drafting itself is startlingly expensive on this stack. Numbers below.

Approach and Choices

Target model: the same unsloth UD-Q4_K_S (15.36GB) file as part two, with identical server flags (-c 8192 -ngl 999 -fa on -b 512 -ub 512). That keeps yesterday's baseline directly comparable — and today's baseline did reproduce it exactly (6.00 vs 6.00 tok/s).

Draft model: of z-lab's three official GGUF quants I picked Q4_K_M (1.14GB) — it matches the --draft-bits 4 used in the MLX test, z-lab's own eval shows Q4_K_M's acceptance length (5.39) actually beating BF16 (5.28), and part one's retest already established that an 8-bit draft underperforms 4-bit on this machine. Q8_0 (2.06GB) and BF16 (3.86GB) excluded: the main model plus draft already presses against the Metal working-set ceiling on 24GB (the OOM section below is the proof), so there is no case for a bigger draft.

Arms: baseline / n-max 7 (the official README recommendation) / n-max 3 (llama.cpp's default) / n-max 12 (probing the ceiling). Method is the same net-speed protocol: greedy (temperature 0), fixed 400 tokens, read timings.predicted_per_second, prose and code prompts, two rounds each.

Mounting: local files use explicit -md. If you pull the main model with -hf, recent llama.cpp auto-discovers and downloads the sidecar by its dflash- filename prefix (find_best_dflash in common/download.cpp) — the same mechanism as mtp- and eagle3- sidecars.

# draft model, 1.14GB
curl -L -o ~/llm/mtp/models/Qwen3.8-27B-DFlash2-Q4_K_M.gguf \
  "https://huggingface.co/z-lab/Qwen3.8-27B-DFlash2-GGUF/resolve/main/Qwen3.8-27B-DFlash2-Q4_K_M.gguf"

# the official README invocation (spoiler: OOMs on a 24GB machine)
./build/bin/llama-server -m Qwen3.8-27B-UD-Q4_K_S.gguf \
  -md Qwen3.8-27B-DFlash2-Q4_K_M.gguf \
  --spec-type draft-dflash --spec-draft-n-max 7

Test Procedure

The benchmark script reuses part two's scaffolding (with all its pitfall fixes baked in: curl -o writing responses directly, pkill -INT matching by process name, /health body check). Four arms run sequentially; each starts and stops its own server under /usr/bin/time -l for peak memory:

Every request in the n-max 7 and n-max 12 arms returned {"error":{"code":500,"message":"Compute error."}}. The server stderr shows Metal running out of memory:

The load-time log had already warned: ggml_metal_log_allocated_size: warning: current allocated size is greater than the recommended max working set size. 15.36GB of main model plus 1.14GB of draft equals 16.5GB of weights — past the Metal recommended working set on a 24GB machine. Larger n-max means wider verification batches and bigger compute buffers; 7 and 12 push it over the edge.

Was the OOM just ambient memory pressure? During the first run the system had only 314MB free, so that suspicion deserved a controlled retry. With free memory recovered to 4.5GB:

• n-max 7 + -c 8192: still OOMs on every request — it's inherent to the configuration, not the environment;
• n-max 7 + -c 4096 (halving context to buy headroom): the first round completed (prose 1.73, code 3.32 tok/s), then the second round OOMed again — running this close to the ceiling doesn't survive consecutive requests.

Results

All configurations that produced numbers (400 tokens, greedy, net generation speed):

For reference, the MLX stack on this same machine: baseline 6.43–6.51, DFlash 2 at block-size 5 does 10.8–12.2 tok/s (1.66–1.88x), block-size 3 does 11.6 (1.79x). Same algorithm, same machine — one stack gets 1.8x, the other gets 0.5x.

The arithmetic that closes the case

The code prompt's acceptance rate is 83.8%, landing a mean of 3.5 tokens per step (mean len = 3.50 in the server log) — by MLX experience that should be worth 1.5x or more. Why is it still 23% underwater? Divide total time by the number of verification steps:

• Prose: 133.9s / 174 speculative steps (400 − 226 accepted) = 0.77s per step
• Code: 86.3s / 115 speculative steps (400 − 285 accepted) = 0.75s per step

Each speculative step (one draft-block generation plus one batched verification) consistently costs 0.75–0.77 seconds, while a baseline token takes 0.167s. One step costs the equivalent of ~4.6 ordinary tokens, and at n-max 3 a step's maximum yield is 4 tokens (3 drafts all accepted plus the verification token). Even at 100% acceptance, this stack loses money: 0.77 ÷ 4 = 0.19 s/token, worse than 0.167. Same story at n-max 7: 1.44s per measured step, costing 8.6 tokens against a ceiling of 8.

That 0.77s has two components: the unamortized verification batch (the 1.13x measured in part two) plus the cost of drafting itself — a 1.1GB block-diffusion model doing a full forward pass plus candidate-path selection every step, with no Metal-specific optimization behind it. Splitting the two precisely would need a profiler, but the conclusion doesn't: cost ≥ maximum yield is a structural dead end that no tuning escapes.

Two reproduction details worth recording:

• Greedy consistency: the code output is byte-identical to baseline; the prose output diverges at character 1294 — the exact same position where mtp4 diverged in part two. That confirms the phenomenon as systematic floating-point divergence between different Metal kernel paths at near-tied greedy picks, unrelated to which speculative algorithm is running.
• The official numbers aren't wrong: z-lab's README acceptance lengths (5.1–5.4) come from a CUDA GSM8K eval at high reasoning effort, and the README makes no speed claims for Metal anywhere. The gap between "support was merged" and "it's fast on your backend" is exactly what you have to measure yourself.

How to use this

• To accelerate Qwen3.8 on a 24GB Mac: use DFlash 2 on the MLX stack (block-size 3–5). Across three benchmarks it's the only route with a positive return;
• On the llama.cpp Metal stack, don't enable any speculative decoding right now (both draft-dflash and draft-mtp measured as net losses); plain 6.0 tok/s is its optimum;
• This verdict is Metal-specific: CUDA/ROCm users have community-measured gains on both routes — don't let this article talk you out of them;
• If llama.cpp's Metal batched decoding ever improves (watch for batched-bench's batch-8 ratio moving well past 1.13x), this article needs a retest — that's a trackable, quantitative signal.

Pitfalls

1. The official README's recommended configuration OOMs outright on a 24GB machine. --spec-draft-n-max 7 with default context, on top of 16.5GB of weights, crosses the Metal working-set ceiling and 500s every request. The load-time warning allocated size is greater than the recommended max working set size is not noise — when you see it, expect high n-max values to blow up.
2. -c 4096 only buys you one round. The reclaimed space survived the first request pair, then round two OOMed. A configuration living at the memory ceiling is not usable for continuous serving — "it ran once" is not "it works."
3. Peak RSS from /usr/bin/time -l is meaningless under memory pressure. This run's baseline showed a 10.9GB peak versus 15.4GB for the identical configuration yesterday — the missing 4.5GB is mmap'd weight pages the OS evicted. To compare memory footprints, benchmark on a quiet machine or read the Metal allocation log instead.
4. script -a (append mode) fails with tcgetattr/ioctl: Operation not supported on socket when run without a TTY. Creating a fresh session file with script -q file cmd works in the background; append mode doesn't. The retry runs collected output via tee instead.

That closes the trilogy: on this machine, speculative decoding's outcome was never about draft quality (43.8% and 83.8% acceptance rates meet the same fate) — it's about the fixed cost per speculative step on the backend. Two signals worth tracking next: llama.cpp improving Metal batch amortization, and an MLX mounting path for the native MTP head. Either one lands, and the 2×2 grid gets redrawn.