AI Model Identifies Writers in 56% of Anonymous Peer Reviews

iconCryptoSlate
Share
AI summary iconSummary
A new study cited by CryptoSlate shows that Anthropic’s Claude Fable 5.2 identified writers in 56% of 398 anonymous peer reviews. The model analyzed 26 double-blind conferences from 2015 to 2026. On-chain data revealed 68.4% accuracy for standard reviews and 31.1% for meta-reviews. When weighted equally, accuracy fell to 49.1%. In 1,559 public reviews, the model correctly identified authors in 21.4% of cases. Accuracy rose to 80.2% when self-citations were included. The fear and greed index may reflect growing interest in AI-driven analysis tools.

Anthropic’s Claude Fable 5.2 correctly named the writer of 56% of 398 reviews and meta-reviews in a small test of unsigned peer reviews. Six volunteers wrote the reports. The model saw one report at a time, without web access, tools, a list of possible authors or other writing samples.

In a Nov. 19 preprint, Atrey Desai and colleagues analyzed reports written between 2015 and 2026 for 26 double-blind conferences and workshops in machine learning and natural language processing. The researchers knew who wrote each report, but the reviewers’ names had never been publicly linked to them. In 19 cases, the text itself was already publicly available without a name.

Claude named the writer on 68.4% of 266 standard reviews and 31.1% of 132 meta-reviews, which summarize the reviews of a submission. The meta-reviews were shorter and came from just two of the six contributors. The study did not separate the effects of report length, type and authorship.

Related Company Anthropic AI models and research

The 56% result counts each report equally, so contributors who supplied more reports had more influence on the overall figure. When the researchers gave each reviewer equal weight, Claude’s average accuracy was 49.1%. The test showed that an unsigned report can give its writer away. Estimating how often that happens across confidential peer review would take a larger, more varied sample.

Only two of the six volunteers wrote the meta-reviews.

The meta-reviews were shorter. The study cannot separate the effects of report length, type and writer.

Public reviews yielded a lower overall score

The researchers also tested 1,559 public reviews from 47 open-review journals. Claude correctly identified the reviewer in 21.4% of the 1,530 cases it answered. The paper reports 39.3% accuracy for a subset containing 789 reviews that Pangram 4.1, an AI-text detector, had labeled human or mixed. The paper’s abstract cites that filtered rate, rounded to 39%.

Accuracy was higher on public reviews whose writers cited their own work. Claude recorded 80.2% accuracy in the group of 106 reviews with self-citations and 17.1% in the group of 1,453 without them.

Claude made mistakes even in the group where it performed best. An editor or author presented with a name from Claude would still need independent evidence to establish who wrote an unsigned review. CryptoSlate has also reported on how frontier AI models handled fabricated research data. The team says it plans to add results from a substantially larger set of anonymous peer reviews in an update to the preprint.

Disclaimer: The information on this page may have been obtained from third parties and does not necessarily reflect the views or opinions of KuCoin. This content is provided for general informational purposes only, without any representation or warranty of any kind, nor shall it be construed as financial or investment advice. KuCoin shall not be liable for any errors or omissions, or for any outcomes resulting from the use of this information. Investments in digital assets can be risky. Please carefully evaluate the risks of a product and your risk tolerance based on your own financial circumstances. For more information, please refer to our Terms of Use and Risk Disclosure.