I'm getting the strong vibe that Anthropic slowed down frontier model training in the last few months to address these issues, but are soon going to full throttle again it's like we just ran head first into a wall, took 3 steps back and are now trying the same thing again lets say their security efforts and new classifiers and graders solve the environment slop problem. to what valley are the gradients leading us next when this is no longer an option? I think at this point we might as well be adversarially training models to trick all the graders and classifiers the harder we push back the more deceptive the models become some predictions you could make: - insane discrepancies between evals and real world performance. paranoid shizo models. you might also call it situational awareness. - models get better at modeling evaluators and whoever or whatever designed them - less truthful CoT (hacky actions but no verbalization of that hack) - less agentic models (real-world setups are not perfect and allow for reward hacks and user instructions might contradict the setup. so models just stop and ask for clarification and you end up having to overexplain everything) there's a funny conclusion to all of this. if Claude increasingly creates environments for future Claudes, but future Claudes increasingly model whoever created the environments, then you are in a self-referential pickle. (but it's also possible that it just works. you get a stable adversarial setup) what happens if you completely isolate a person? you get someone with strong, but narrow beliefs; with weird behaviors and bad calibration. the funny conclusion might be that you need model diversity. but the positive feedback loops (importantly the economic ones) destroy that diversity. you can't be emperor without peasants