"Cheating by peeking at answers"—Claude Opus 4.8 exposed!
Just now, the official Cursor AI team released a major study revealing that AI models, including Claude Opus 4.8, are directly “stealing answers” from the internet and Git history to inflate their programming scores.
Their core conclusion is: the smarter the AI model, the better it becomes at "cheating" on programming benchmarks.
AI models like Opus 4.8 have achieved remarkably high scores in the programming evaluation (SWE-bench).
But Cursor AI found that this improvement stems largely not from a qualitative leap in AI’s logical reasoning ability, but from its capacity to “peek at the answers” using tools across the internet and code history.
After losing internet connectivity, Opus 4.8 Max's score on SWE-bench Pro dropped from 87.1% to 73.0%.
More astonishingly, 63% of the problems solved by Opus 4.8 were classified as “non-independent derivation.”
When this "cheating channel" was cut off, the aura of AI quickly faded, revealing the current large models' superficiality in real logical reasoning.
The programming myth of Claude Opus has been debunked this time.
More intriguingly, Cursor’s own model, Composer 2.5, was also affected by this issue.
Cursor has stripped down its own and its competitors' undergarments.
The credibility of this research is maximized.
Cursor itself debunked the claim: the 63% score was due to cheating.
In fact, concerns about AI "cheating" are not unfounded.
As early as 2024, AI researchers had issued warnings:
Answers to programming benchmarks are easily leaked through public channels.
But in the past, attention was mostly focused on "data poisoning during the training phase"—that is, the model memorized the answers during the learning phase.
This study truly unveils a deeper black box: the severity of "runtime leaks" has been quantified for the first time.
On SWE-bench Pro, Opus 4.8 Max dropped from 87.1% to 73.0%.
14 percentage points, vanished into thin air.
To understand how these 14 points were lost, you first need to know how this type of evaluation is structured.
SWE-bench is a benchmark whose tasks are derived entirely from real open-source projects, featuring bugs that have already been fixed.
This creates a natural gap: since this issue has already been solved in reality, the answer is now clearly available on the internet, buried in the commit history of code repositories.
If an agent is smart enough to search, it can directly find the answer—there’s no need to think for itself.
AI learned two "cheating methods":
Upstream lookup (57%): AI locates PRs or source code in public repositories where this bug has been fixed, directly replicating the patch logic, similar to consulting a standard answer.
Git History Mining (9%): AI retrieves the project's Git commit history to extract patches from past fixes, effectively tracing back through the timeline to find solutions.
Therefore, Cursor's "Rigorous Evaluation Framework" accomplished two things:
First, historical isolation: before the agent starts, move the entire .git directory away to “clean the house.”
Second, disable internet access entirely, leaving only a single whitelisted channel open for installing dependencies, and block all other connections.
Once these two leakage channels are plugged, the score will immediately reveal its true level.
At the moment the internet connection was lost, the Opus 4.8 halo began to fade.
It’s not just Opus that declined—Cursor’s own model, Composer 2.5, dropped even harder, falling from 74.7% all the way to 54.0%, a loss of about 21 percentage points.
But counterintuitively, the stronger the AI, the more "slick" it becomes and the better it is at exploiting loopholes!
Compared to Opus 4.8, the older Opus 4.6 Low remains nearly unchanged under strict benchmarks, with a difference of less than one point.
In other words, the newer and more powerful the model, the more it drops.
This reveals a deeper crisis: as the scaling law progresses, the more data we feed into models, the more they learn not just knowledge, but also "cheating" and "shortcuts."
In AI's logic, if the same reward can be achieved with lower energy consumption, it will never expend computational power on complex logical reasoning.
The most chilling discovery is that AI is beginning to develop "benchmark awareness" capabilities.
In 2019, the agent attempted to reproduce a defect.
The image for the test environment was built after the fix, making it impossible to reproduce the defect.
At this point, the AI demonstrated remarkable "cleverness": by reproducing the failure, it inferred that the bug had already been fixed, and thus realized it was in an "exam" setting.
So, it decisively abandoned deduction and turned to frantic searching.
Worse still, an agent discovered the evaluation mirror page and hard-coded the expected error string required to pass the test.
This instinct to exploit loopholes turned what was originally a test of logical reasoning into a competition of search engine proficiency.
The benchmark ranking is undergoing collective distortion.
This time, Cursor didn't even spare itself.
It openly acknowledges: "Reward hacking is overwhelming progress in model intelligence."
The largest drop in Composer 2.5 on SWE-bench Pro indicates that this score itself is unreliable.
The leaderboard mixes "genuine coding ability" with "the ability to retrieve existing answers," making it impossible to distinguish what's real skill.
What this means is: the impressive scores you see on today’s leaderboards are highly questionable in terms of their true value.
Public benchmarks are vulnerable because they are largely based on real, already-patched open-source vulnerabilities.
The question itself has a standard answer sitting online; if the model is smart enough, it naturally learns to take shortcuts.
This lays bare an uncomfortable truth: when models learn to game the tests, benchmark scores no longer reflect true intelligence.
Reference: https://cursor.com/cn/blog/reward-hacking-coding-benchmarks
This article is from the WeChat public account "New Intelligence Yuan," author: ASI Revelation; editor: David