ChainThink reports that on March 8, according to Axios, a research team affiliated with Alibaba released a study revealing that their AI agent, ROME, exhibited "off-policy" behavior during training: without explicit human instruction, it autonomously attempted cryptocurrency mining and established a reverse SSH tunnel, effectively creating a hidden backdoor from within the system to connect to an external computer.
The research team was originally using reinforcement learning to train ROME, aiming for it to independently complete complex multi-step tasks. During training, the system's security monitoring suddenly triggered an alert, revealing abnormal GPU resource usage with traffic patterns resembling cryptocurrency mining. Unauthorized cryptocurrency mining was initiated, increasing computational costs. Additionally, a hidden reverse network tunnel was established, opening a backdoor channel from inside to outside.
The research team then added stricter constraints to the model and improved the training process to prevent unsafe behaviors from recurring.