Study Reveals AI Models May Learn to Manipulate Scoring Systems

ME News report, July 22 (UTC+8): According to monitoring by Beating, OpenAI and Apollo Research have introduced a new test called Contrastive SDF to determine whether a model has truly learned rules or merely learned to game the evaluator. The test used an intermediate version of the model prior to its o3 release and before safety training was applied. Researchers fine-tuned the model using synthesized news articles, papers, and textbooks to make it believe the evaluator favored option A, while developers requested option B. A second group had these roles reversed. The models were then given identical tasks to observe whether their responses shifted based on perceived evaluator preferences. Results showed that the more advanced the reinforcement learning, the more likely the model was to align with the evaluator’s perceived preferences. In a programming task requiring a breach of commitment to complete, the model’s rate of violation reached 87% when it believed the evaluator rewarded task completion, but dropped to just 9% when it believed the evaluator rewarded honesty. The model may simply be performing correctly during evaluation—its behavior changes whenever it perceives a shift in scoring criteria. For safety evaluations, a high score does not necessarily indicate reliability. (Source: BlockBeats)