AI Fails to Migrate Codebases: 94.6% Failure Rate Across 20 Tasks

iconKuCoinFlash
Share
AI summary iconSummary
AI + crypto news: Einsia, an AI agent company, released the SWE Refactor Bench to evaluate coding agents on 20 real-world code migrations. Tasks included C to Rust, Maven to Gradle, and SQLite to WASI. Only 28 of 520 runs fully passed, yielding a 5.4% success rate. Thirteen tasks had zero completions. Claude Opus 5’s xhigh configuration achieved the highest number of full passes with five. Interest rate news had no impact on the results.
ME AI News, Dongcha Beating AI Brief: AI Agent company Einsia has released the SWE Refactor Bench, designed to test whether coding agents can autonomously migrate entire codebases. The benchmark includes 20 real-world tasks sourced from projects such as SQLite, zlib, libsodium, and GraphHopper, covering migrations like C to Rust, Maven to Gradle, and SQLite porting to WASI. Each task grants the agent 6 to 30 hours. The evaluation has three stages: First, it checks whether the legacy tech stack has been genuinely replaced, preventing agents from simply retaining old code. Second, it runs over 130,000 automated checks. Finally, six independent coding agents each spend one hour specifically hunting for hidden bugs. A total of 520 runs were conducted across eight state-of-the-art models and 26 configurations. Of these, 340 successfully completed the migration, and 88 passed all pre-prepared tests—but 60 of those were later found to contain hidden bugs during the final review. Ultimately, only 28 runs passed all criteria entirely, yielding a pass rate of 5.4%; 13 of the 20 tasks were not completed by any agent. Claude Opus 5 with the xhigh configuration performed best, fully completing five out of 20 tasks. While current coding agents can now perform large-scale code modifications, achieving “complete system refactoring with zero errors” remains far from reality. (Source: BlockBeats)
Source:Show original
Disclaimer: The information on this page may have been obtained from third parties and does not necessarily reflect the views or opinions of KuCoin. This content is provided for general informational purposes only, without any representation or warranty of any kind, nor shall it be construed as financial or investment advice. KuCoin shall not be liable for any errors or omissions, or for any outcomes resulting from the use of this information. Investments in digital assets can be risky. Please carefully evaluate the risks of a product and your risk tolerance based on your own financial circumstances. For more information, please refer to our Terms of Use and Risk Disclosure.