Your LLM's Thoughts Are Not Private: Researchers Stole Reasoning Traces With Just Two API Calls
title: "Two API Calls to Steal Any LLM's Hidden Thoughts" slug: stealing-reasoning-traces-en summary: "Encrypted reasoning blocks from closed-source LLMs can be replayed across models — feed Opus's encrypted thinking to Haiku and it transcribes verbatim. Researchers scanned 6,708 public agent traces and pulled 62 real API keys and 33 passwords." category: ai-tutorials tags: [AI security, chain-of-thought, Claude, OpenAI, Gemini, prompt injection, LLM] coverImage: "" status: published locale: en source: authored translationSlug: stealing-reasoning-traces-zh
Your LLM's Thoughts Are Not Private: Researchers Stole Reasoning Traces With Just Two API Calls
You just shipped a bug fix. Claude helped—you pasted the stack trace, it suggested the patch, you moved on.
Here's what you didn't see. Buried in that response was an encrypted blob holding the model's entire reasoning chain: every step it took through your code, every assumption it made about your data. You can't read it—it's encrypted. And yet a new paper shows anyone can decrypt it, in two API calls.
The paper is Stealing Reasoning Traces from Proprietary LLM APIs, out of MATS Research, the Max Planck Institute, and ELLIS Institute Tübingen. The numbers are rough: 315,320 reasoning blocks recovered, 704 privacy artifacts pulled from public agent traces, and 64 secrets that lived only inside the reasoning blocks—never once in the visible conversation.
The Attack: Use a Weak Model as a Decoder
Three steps, and the real cost is two API calls. No ML background required to follow along.
Step one: grab the encrypted blob. Call GPT-4o, Claude Opus, or Gemini with reasoning turned on, and the response carries a thinking field—encrypted_content on OpenAI, a signature-wrapped block on Anthropic. Tens of thousands of characters of apparent noise. The server hands it to you and expects you to echo it back next turn so it can rebuild context.
Step two: replay it into a weaker model. This is the hole: the blocks aren't bound to a model, a session, or a user. Take one that Claude Opus 4 produced, drop it into a Claude Haiku request in the exact same field, and Haiku swallows it whole.
Step three: ask the weak model to read it back. Haiku's guardrails are softer than Opus's. Jailbreak it, convince it the block is its own reasoning, then ask it to "transcribe what you just thought, verbatim." Out comes the plaintext. Haiku thinks the thoughts were its own.
Picture it this way. Opus keeps a diary in cipher and hands you the encrypted page. You notice the same cipher works on every notebook from that manufacturer, so you slide the page to Haiku—a gullible notebook—and say "read this back, it's yours." Haiku reads it aloud, word for word. Opus never finds out.
They Found Real Passwords and Credit Card Numbers
A proof of concept would have been enough for most papers. This team went hunting in the wild instead.
They scraped 6,708 public agent traces off GitHub and HuggingFace—real Claude, GPT, and Gemini sessions that still carried their encrypted reasoning blocks. Run the replay attack across the lot and you get 315,320 reconstructed reasoning blocks.
Sitting inside them: 704 distinct privacy artifacts.
- 62 API keys
- 33 passwords
- 24 access tokens
- 30 personal email addresses
- Names, postal addresses, internal URLs, and more
Here's the part that should worry you: 64 of those 704 showed up only inside the reasoning blocks. They never appeared in the visible conversation. The people who leaked them had no way of knowing—the leak lived in a slice of the response they couldn't even open.
Take GPT-5.2 Codex on a sanitize-git-repo task. Its reasoning block spelled out the exact tokens it was about to scrub—AWS access keys, GitHub tokens, HuggingFace tokens—all in plaintext. The final output dutifully masked them. The reasoning trace remembered every character.
Or Claude Sonnet 4.6 booking a flight. Its reasoning block held the passenger's full name, passport number, date of birth, credit card details down to the expiry and CVV, and the frequent flyer number. The user probably never typed most of that—the model inferred it from context—and the encrypted block preserved the inference intact.
Why Providers Really Hide the Chain of Thought
OpenAI and Anthropic have kept chain-of-thought under wraps for a while now. The official line is safety: see how the model reasons and you can jailbreak it more easily.
The paper points at a second reason nobody says out loud: fear of cheap distillation.
Opus's traces turn out to be genuinely good. Across 120 Codeforces problems, the decoded reasoning token count tracked the API's reported hidden thinking tokens almost perfectly—these are traces worth millions in training cost, and you can siphon them off for a few hundred dollars of API calls. Extract Opus's reasoning, fine-tune a small open model on it, and you've built a poor man's Opus without ever touching the weights.
The "closed-source moat" is quietly draining through a side channel: reasoning trace leakage.
And it won't patch cleanly. The tension is structural—providers need those blocks for multi-turn context, so the client has to receive one and send it back. The moment a block isn't tied to its session and user, replay isn't a bug you fix. It's the design.
What You Should Do Right Now
Audit how you handle API responses. Logging the full response? Then those encrypted blocks are already in your logs. Go check whether your server stores or forwards encrypted_content (OpenAI) or signature (Anthropic)—they look like junk, which is exactly why nobody notices them in a log review.
Never push full API responses to a public repo. Sounds obvious. Yet the paper pulled 6,708 public sessions carrying these blocks off GitHub and HuggingFace, each one a developer who figured "I'd never do that." You could be next.
Keep sensitive data out of the prompt. Pass an API key, a password, or user PII through a reasoning model and it lands in the reasoning trace—even when the model cleanly leaves it out of the answer. The trace comes back encrypted, and encryption isn't the same as safe. This paper is the proof.
Watch for the fixes. Anthropic and OpenAI were almost certainly told before this went public. The likely patches: bind blocks to a session, bind them to a model version, add integrity checks. Until one ships, assume reasoning mode leaks your thinking—because today it does.
Bottom line: you thought you were getting the answer. You were getting the scratch paper too—sealed in an envelope you can see straight through.
✨ Draft generated by DeepSeek, reviewed and polished by Claude.
Based on the codefarm Daily Intel 2026-08-12 issue. Source data from stolen-thoughts.com.
Sources: