Author: Claude, Deep潮 TechFlow
Original title: Latest paper sparks debate: AI "deep thinking" process can be freely distilled; closed-source companies' most valuable training assets are being emptied out
DeepChain Summary: Every time you ask AI a question, it first "thinks deeply" in the background before responding—this unseen thought process is the core competitive advantage of OpenAI and Anthropic. Now, a group of researchers has publicly revealed a method to extract this entire thought process. Along with it, they’ve also exposed credit card numbers, passwords, and email addresses users pasted in. This isn’t a distant security paper—it’s a harbinger of how your future interactions with AI may soon change.
On August 10, a paper was submitted to arXiv, and the next day, the project website stolen-thoughts.com went live, displaying verbatim "thought logs" extracted from multiple proprietary models, reaching 500 points on Hacker News.
Project lead Alexander Panfilov wrote on X: "We found a way to extract hidden reasoning from state-of-the-art models by exploiting vulnerabilities in the APIs of all leading AI companies."
In other words: you thought only AI knew what it was thinking, but in fact, someone can lay it all out for you.
What you feed into AI may be leaking out along with its “thoughts.”
Researchers scanned approximately 7,000 publicly shared AI assistant conversation logs, decrypted the enclosed "thought processes," and discovered things that should not have been there.
Panfilov said in a tweet: "We initially scanned about 7,000 public sessions and found 62 unique API keys, 33 email addresses, 33 passwords, and other sensitive information."
More alarming are the details: within the "thought process" of a flight booking task lie full names, email addresses, passport numbers, dates of birth, and credit card numbers with security codes. Keys from platforms such as Anthropic, AWS, and GitHub also appear in these examples. In other words, the credentials you provide to help the AI do your work are saved along with its thought process—and can then be stolen by others.
"If you have ever shared online Claude Code or Codex sessions with encrypted reasoning blocks, they can both be decoded, exposing your personal data," wrote Panfilov.
The most direct reminder for regular users: Don’t paste sensitive information into AI, even if it says, “I won’t leak it.”
The vendor first says "It's fine," then secretly fixes it.
This incident did not happen suddenly. In May of this year, Matthew Green, a cryptography professor at Johns Hopkins University, had already reported a similar vulnerability to the vendor, who responded that they saw no security impact.
Once the Panfilov team made a formal disclosure, the vendor’s stance changed. “We then followed the responsible disclosure process, and the vendor has patched several issues caused by this vulnerability, which, to my knowledge, are still ongoing,” said Panfilov. The paper also confirms that after disclosure, the researchers were no longer able to reproduce the same attack.
The issue was that the vulnerability existed for months without users’ knowledge. It was only after it was made public and discussed that the fix was finally implemented. This isn’t the fault of a single company, but rather the first time the industry’s assumption of “crypto equals security” has been publicly exposed. For readers, the key takeaway is this one sentence: The AI company you trust may not have told you about all the risks.
The model you're using may no longer be as 'exclusive' as before.
Longer-term changes at the industry level.
Reasoning ability is the foundation of OpenAI, Anthropic, and others’ pricing, and also the part they are least willing to reveal. Once this reasoning can be extracted at scale, competitors could use it at extremely low cost to teach their own models the “thought processes” of the strongest models. The paper also mentions an unreviewed preliminary observation: using a small amount of the strongest model’s “reasoning” to guide another model clearly pulls the latter’s responses toward the former’s direction.
What does this mean? The moat of closed-source models was once “you can’t buy my intelligence with money.” Now, this wall has developed cracks. For users, this isn’t necessarily bad in the short term: stronger competitors may emerge faster, and prices could be driven down. But the cost is that you can no longer tell whether a model is truly intelligent or simply copying someone else’s ideas.
Who truly owns “thoughts you pay for but cannot see” is the essence of this debate. A technical fact is already clear: as long as this thought remains in the client’s hands, encryption is merely an illusion.
For closed-source vendors, what’s harder to fix than vulnerabilities is the narrative: the story that “reasoning is a moat” has been proven, for the first time, to be bulk-disassemblable.
Twitter: https://twitter.com/BitpushNewsCN
BitPush Telegram community: https://t.me/BitPushCommunity
BitPush TG subscription: https://t.me/bitpush