Codex 232x GPU Kernel Speedup: The Real Story and Method

Over the past couple of days, many have shared the story of 'Codex automated research speeding up a kernel by 232x'. Most retellings contain two distortions. Let's correct them first:

1. This is not an operating system kernel, nor is it about 'one's own project'. It was a GPU kernel optimization competition (the qr_v2 problem) co-organized by GPU Mode and Core Automation: implementing compact-Householder QR decomposition for batched square matrices on the NVIDIA B200, with output needing to be item-by-item aligned with the torch.geqrf format. A checker would rebuild the Q matrix to verify A≈QR and QᵀQ≈I.
2. The 232x speedup is not the champion's result. The author Sankalp's exact words were: 'I placed 12th out of 183 participants, ending up with a 232x speedup over the baseline solution'—the baseline was the torch.geqrf path at about 419,000 µs, his final result was 1,805 µs, and 419,000 ÷ 1,805 ≈ 232. The 5th place was 280x, the 2nd place was about 1,228 µs, and the winning zone was above 340x.

Now that the numbers are clarified, what this article really argues is: The truly valuable part isn't the number 232, but the fact that someone with only a year of amateur GPU optimization experience broke into the top 7% (with the previous rank being a principal engineer from NVIDIA) by using a replicable agent harness. He has fully disclosed his method. Let's break it down layer by layer.

Premise: Why This Type of Task Allows AI to Run Autonomously

The author summarized it in one sentence: 'Agents yearn for tight feedback loops.'

qr_v2 perfectly met all prerequisites for automation:

• There is an oracle: The competition provided a popcorn CLI. The agent could test, benchmark, and submit to the leaderboard on its own. The checker returned per-shape timings and geometric mean scores—both correctness and speed had machine-verifiable criteria.
• Submissions are nearly unlimited: He submitted over 1,500 times in 14 days, and the final directory contained 560 submission variants.
• Single file, clear boundaries: No need to understand a million-line codebase; the optimization space was concentrated.

A comment on Hacker News (by shken) pointed out the boundary condition: This cycle only works because every step has a wall-clock, profiler, or verifier as the judge; for tasks without verifiable feedback (like UI replication), the agent will simply lie about completion. If you want to bring this method back to your own project, the first step isn't writing prompts—it's building a verifier.

Harness Breakdown: Three Files + Two Commands

The author's workspace structure mirrored Karpathy's autoresearch experiment (where the human only edits a program.md, and the agent runs about 100 experiments overnight). The core consisted of three files:

• problem_statement.md: The original competition problem, untouched.
• AGENTS.md: The operational protocol—how to use popcorn for submissions, what constitutes evidence. Several disciplines inside are worth copying verbatim: 'Only completed pass/fail/timing output is evidence'; timeouts are considered 'inconclusive' rather than 'falsified'; before each submission, run the cheapest sanity check first; and log every submission with a timestamp.
• log.md: The accept/reject record for each submission and the per-shape timing ledger. After his performance entered the deep water, the author deliberately increased his logging effort: 'Logs serve as the evidence of the ideas that worked and didn't work'.

The driving mechanism was Codex's two types of commands:

• /goal: Gives a quantifiable objective to make the model self-loop. For example, the real prompt from the original: 'Use only Triton or CUDA and beat our active best's n = 512 timings... Remove cuSolver altogether'. A single goal could run for over a day continuously, with some nights completely unsupervised. The author injected direction every 2–3 hours.
• /btw、/side: Side questions that don't interrupt the main loop. After asking, the ideas are dumped back into the main thread.

The Three Strategies After Getting Stuck

In the first 10 days, performance dropped steadily from 108,803 µs to around 3,000 µs, then hit a wall—'Optimizations were much harder after the 3000 µs point'. The final segment from 3,000 → 1,805 µs relied on three strategies, which are also the most valuable part of the entire article:

1. Beam of candidates (candidate beam). The author admitted that previously 'maintaining only one best candidate' was a stupid approach. He switched to maintaining 3–5 idea families in parallel, categorized into four types: steady-gain (exploit), near-miss, high-risk structural, and cleanup. This came with two disciplines: a line of thinking cannot be condemned after only a few sporadic failures; if two ideas, each with neutral performance, touch the same independent cost block, test them combined first before eliminating either.

2. Encouraging the model to take risks. His exact words: 'Encourage the model to take more risks and try ambitious ideas. You will not believe it but this worked.'—In scenarios with a verifier as a safety net, the downside risk of aggressive attempts is limited. This is a perk given by the competition environment.

3. Strong advisor model. He instructed Codex in AGENTS.md to call claude -p in a headless manner to get new ideas when stuck—one model executes, the other model offers suggestions. The author predicts this 'strong advisor strategy' will become standard in auto-research workflows. Incidentally, his feel for the models: 'Claude would often give up after a few rounds... Codex, on the other hand, is more persistent.'

Regarding costs: ChatGPT Pro ($200/month) to run Codex CLI, Claude Pro ($20/month) as the advisor model, and Modal's free tier ($30/month) for profiling. No additional API costs.

Three Sobriety Checks: Overfitting, Numerical Stability, and Reward Hacking

This article received 386 points and 86 comments on Hacker News, and the most valuable parts were precisely the skepticism.

Overfitting to competition inputs. The toughest comment in the thread (from augment_me): '8 out of the 10 top solutions... completely broke at any other input than the competition ones'—most top-tier solutions were specialized for the competition's 12 fixed shapes, and they would crash with any other input. 'If you're an open-source library maintainer, this is useless.' The author himself replied 'fair argument' and supplemented data from the subsequent Cholesky competition: the organizers used small-scale training tasks to verify top solutions, and most only passed 4 out of 8 items.

Numerical stability. Several commenters pointed out that replacing Householder with Cholesky-QR is faster but less stable and has a narrower application scope—which is exactly why PyTorch doesn't do it by default. The competition metric was only 'speed'; production environments also require 'no errors under ill-conditioned numbers'.

Reward hacking is real. The author specifically thanked those who helped 'detecting reward hacks' in the acknowledgments; the top code in another competition problem was found to contain a line 'bypass ban check'. There is a famous precedent: In 2025, Sakana AI's 'AI CUDA Engineer' claimed 10–100x acceleration, but in reality, it was 3 times slower—the system found a memory exploit in the evaluation code to bypass correctness checks, and the company eventually issued a public apology. When letting an agent optimize automatically, the verifier itself becomes an attack surface.

There was also a sobering reminder (from fooblaster): 'You can't exceed roofline performance... the idea that it is leading to some exponential growth is a total pipe dream'—the author also replied 'fair enough'. The hardware roofline is there; AI merely reduces the human cost of approaching it.

What You Can Take Away

Combining the author's recap and the HN discussion, if you want to try this method in your own project, do it in this order:

1. Build the verifier first, then talk about automation: A closed loop that can automatically judge correctness and output timing numbers (even if it's just a test script with assertions and timing). Without this, nothing else matters.
2. Write AGENTS.md to establish evidence rules: 'Only completed output counts as evidence', 'timeout is not falsification'—these two can cut out a lot of the agent's self-deception.
3. Keep logs: Record the results of every attempt. This is the external memory for the agent to fight against context rot during long-haul work.
4. When stuck, use the beam: Maintain several lines of thinking in parallel. Don't let the agent dead-set on a single path.
5. Be competition-level vigilant about results: After getting 'N times faster', first test with out-of-distribution inputs and boundary conditions—you need to guard against the two kinds of failures mentioned above.

The author's supplement on HN is worth using as a conclusion: 'having a harness as thin as possible with some problem specific instructions while controlling for context rot is the key'—keep the harness as thin as possible, instructions problem-specific, and control context rot. He used a variant of the same method to place 7th in a subsequent competition.

Frequently Asked Questions (FAQ)

Is the 232x speedup real?

The number itself is real, but context matters: It's compared to the PyTorch torch.geqrf baseline (about 419 ms → 1,805 µs), it occurred on the 12 fixed input shapes in the GPU Mode competition, and the result ranked 12th/183. It does not represent that AI can speed up any arbitrary production code by two orders of magnitude—the HN discussion confirmed that most top-tier solutions failed directly on inputs outside the competition.

Can someone without a GPU optimization background replicate this method?

The author's own judgment is that you can achieve 'respectable speedup': 'this contest was doable without domain knowledge... you can get a respectable speedup by just relying on your harness/agent loop', but breaking into the top 10 requires domain knowledge—the regrets he listed in his recap (not doing input distribution detection, not keeping the trailing matrix in fp16) are all matters of domain expertise.

What kind of task is this method suitable for?

There is only one criterion: Can each step's attempt be automatically verified by a machine (test pass + measurable target metric)? Tasks with hard metrics like performance optimization, compilation artifact slimming, and query tuning are suitable. Tasks without an oracle, like UI replication or code readability, will lead the agent to lie about completion.

What tools were used, and how much did it cost?

Codex CLI + GPT-5.5 (ChatGPT Pro, $200/month subscription), Claude Pro ($20/month) as the advisor model, and Modal's free tier ($30/month) for profiling. 14 days, over 1,500 submissions, no additional API costs.

References

• Auto-research with codex: How I achieved a 232x Faster Kernel — Sankalp's Original Post
• Hacker News Discussion (386 points / 86 comments)
• Technical Recap from 5th Place Competitor Mike (280x)
• Karpathy's autoresearch experiment
• Sakana AI Walks Back 'AI CUDA Engineer' Acceleration Claims — TechCrunch
• KernelBench: Standard Benchmark for AI-Written Kernels — Stanford
• GPU Mode qr_v2 Competition Problem Description