The security team at 1Password conducted an evaluation in which they used six complex vulnerabilities from software such as Linux and Chrome to test ChatGPT-5.5 and Claude Opus-4.8’s ability to generate patches, ultimately assessing 6,080 attempts. The study was published on August 6. The results showed that 26% of the patches successfully fixed the vulnerabilities without breaking existing functionality; 20.1% closed the vulnerabilities but introduced unintended side effects elsewhere in the program. Approximately half failed to fix the original vulnerabilities at all, and some even introduced new ones. Researchers found that when confronted with Spring AI vulnerabilities, the models frequently focused on manipulating just a few specific characters from the attack sample—rendering the attack ineffective while leaving the underlying vulnerable code untouched. With a slightly different input, the risk could easily return. A patch delivered to you may appear effective and have successfully blocked one attack, making it tempting to lower your guard. “The expected value of patches generated entirely by large models without human review is clearly negative,” concluded the paper’s authors, who are concerned precisely with this scenario: unresolved issues coupled with a false sense of security. I believe that when evaluating AI-assisted programming for efficiency, you must account for the costs of review, rework, and potential incidents. Calculating only how quickly code is generated makes the numbers look good—but if you’re saving time on writing code, are you also planning to cut corners on those who spot problems in that code?