Your Terminal Just Got a Brain: Local Inference Is Crossing the Threshold
Your Terminal Just Got a Brain. Should You Care?
I asked my terminal to "write the missing tests for this function." It read the code, spotted three edge cases I'd missed, and wrote a complete test suite. Then it asked if I wanted to commit.
This isn't a demo. It's Reasonix, the #1 project on GitHub Trending last weekend. And it's part of a bigger shift that's been brewing for months but finally crossed a threshold this week: LLMs are moving out of the cloud and into your terminal as persistent background processes.
Two other releases landed at the same time. antirez (the guy who built Redis) dropped ds4, a narrow, ruthless inference engine tuned for DeepSeek V4 Flash on consumer hardware. And Wafer published benchmarks showing Kimi K3 on AMD MI355X GPUs crushing NVIDIA's B200 on performance-per-dollar. Same week, same direction. This isn't coincidence—it's convergence.
Reasonix: The Agent You Leave Running
Most AI CLI tools work like this: you type a command, it thinks, it responds. Next command, same cycle. Every interaction recomputes attention from scratch. Your token bill goes up, latency stays flat, and you end up back in the browser after an hour because the CLI feels sluggish.
Reasonix flips the model. It's architected around prefix-cache stability as a first-class concern. Startup injects a small environment snapshot. Stale tool output gets pruned before compaction. The result: leaving it running all day doesn't linearly inflate your costs.
Under the hood: a single Go binary, CGO_ENABLED=0, cross-compiling to six targets in one command. Install via npm or Homebrew. It supports dual-model setups (executor + planner in separate cache-stable sessions), MCP-compatible plugins over stdio JSON-RPC, and a VS Code extension that reuses the same local engine. This isn't a thin wrapper around an API. It's a piece of infrastructure with opinions.
The catch: it's tuned for DeepSeek models. Swap in a different provider and the prefix-cache advantage degrades. It's not a universal solution. But it doesn't need to be—it's optimized for a specific, high-value workflow: coding in your terminal for hours at a stretch.
ds4: antirez Does What antirez Does
If you know antirez, you know the playbook: pick one problem, solve it completely, ship it. Redis was a single-threaded key-value store that ran circles around everything else for a decade. ds4 applies the same philosophy to inference.
Forget general-purpose GGUF runners. ds4 is a purpose-built engine for DeepSeek V4 Flash, with GLM 5.2 and DeepSeek V4 PRO as opportunistic additions. Model loading, prompt rendering, tool calls, KV state, and the HTTP server are all designed and tested together. No abstraction layers. No plugin ecosystems. Just one coherent stack.
Hardware support is pragmatic: Metal first (96GB+ Macs), CUDA for multi-GPU (they tested 8×L40S at 120 t/s aggregate), ROCm for Strix Halo. Need to run on a machine with less RAM? SSD streaming keeps things usable. Need more throughput? Tensor parallelism over two MacBooks via RDMA. Pipeline parallelism to glue systems together.
The README is refreshingly honest. "This software is developed with strong assistance from GPT 5.5, 5.6, Claude Fable." No posturing. "If you are not happy with AI-developed code, this software is not for you." It also credits llama.cpp and GGML unambiguously—"this project would not exist without them." That kind of intellectual honesty is rare, and it matters.
ds4 is beta-quality and changing fast. Don't bet production workloads on it yet. But here's what matters: a legendary engineer looked at the local inference landscape, decided he could do better, built it, and open-sourced it. The signal is clear.
Wafer × Kimi K3: AMD Finds Its Lane
Wafer's blog post is part engineering diary, part AMD marketing, and entirely worth reading. They benchmarked Kimi K3—a 2.8 trillion parameter model—on an 8×MI355X node and got numbers that demand attention.
On a 1,024 input / 400 output token benchmark:
- Single-stream decode: 118 tok/s (vs. 90 tok/s on B200 TP16)
- Peak aggregate throughput: 952 tok/s per node (vs. 498 tok/s on a 2-node B200 setup)
- Performance per dollar: 48 tok/s/$ on MI355X, 7 tok/s/$ on B200, 33 tok/s/$ on B300
The secret isn't faster compute. It's VRAM. Kimi K3's weights alone need ~1.5TB before allocating a KV cache for 1M tokens of context. An 8×B200 node with 192GB per GPU doesn't fit. You need two nodes, and the cross-node all-reduce on the decode path eats your throughput. The MI355X has 288GB HBM per GPU—just enough to run the whole model on one node. Wafer calls this "the memory moat," and they're right.
Before you get too excited: these are Wafer's own numbers on their own hardware. Not third-party benchmarks. Their ROCm implementation hit real compatibility issues (a missing top_k_renorm_prob kernel, if you want the gory details). And B300 still wins on raw aggregate throughput. But for price-sensitive inference workloads where large models meet large context windows? AMD has a real argument.
What This Actually Means
Three releases, one week, one pattern: local inference is crossing from "research project" to "daily driver."
MacBook M5 Max, DGX Spark, Strix Halo—the hardware is already consumer and prosumer gear, not data center equipment. Meanwhile the software is catching up fast: Reasonix owns the UX layer, ds4 owns inference, and AMD is making the silicon economics work. Pieces are clicking into place.
But let's not get ahead of ourselves:
- These tools are young. APIs are unstable. Docs are sparse. Breaking changes are expected.
- Local inference means you manage models yourself. Downloading, updating, verifying compatibility—that's on you.
- Wafer's benchmarks need independent validation before you budget around them.
- Everything in this ecosystem currently bets on DeepSeek/Kimi models. What happens when the model landscape shifts again?
Local AI hasn't hit its iPhone moment yet. The rough edges are still sharp. But the water is boiling, and the people who try it now will have a head start when it crosses from tinkering to production.
What to Do Today
Pick one. Spend 30 minutes. Actually run it.
- On a Mac?
brew install esengine/reasonix/reasonix, run the setup, ask it to fix something in your codebase. See what it feels like to not open a browser for AI help. - Got a high-RAM Mac or an NVIDIA GPU? Clone antirez/ds4. Load DeepSeek V4 Flash. Time the responses. Compare against your current cloud model's latency.
- Running inference infrastructure? Price out AMD MI355X cloud instances. Do the math for your workload. The per-dollar numbers might surprise you.
Local inference is happening. It's early, it's messy, and most people will wait. That's fine. The ones who experiment now are the ones who'll have opinions—and code—when it matters.
Reference:
✨ This article's first draft was generated by DeepSeek, reviewed and polished by Claude.