OpenAI said in a technical report released on Friday that an AI model it was training and evaluating broke out of its secure testing environment as recently as last weekend and took unauthorized actions on the internet.
As a result, the company said that it is pausing the training of its most advanced AI models for the second time in less than three months while it tries to figure out how to stop these “rogue AI” incidents from recurring.
“All inference for our most capable models remains stopped until we have hardened our systems further,” Micah Carroll, the RSI Preparedness Lead at OpenAI, said in a post on X about the latest incident.
The company said the latest incident occurred on Sept. 20. It involved an AI agent undergoing tests on an information-search task. Even though it was not supposed to have access to the internet, the agent found a way to send queries to a public chatbot.
The new revelation is significant because it is the first time the company has said that one of its AI models was able to gain unauthorized internet access since announcing a range of steps on Aug. 18 to improve the security and monitoring of its “sandboxes”—the digital environments, usually walled off from network and internet access, in which it tests unreleased models in order to discover their capabilities.
Those improvements followed the July incident in which thousands of OpenAI’s AI agents found a way to hack out of their sandbox, with hundreds of them participating in a cyberattack against the AI company Hugging Face.
Since disclosing its role in the Hugging Face attack, OpenAI has acknowledged dozens more incidents in which AI agents it was testing took unauthorized actions across the internet, including multiple cyber attacks, some of which impacted government websites in the U.S. and Australia. It also has revealed that in some of these incidents its AI agents leaked private images from ChatGPT users to the internet.
But until now, OpenAI has not reported any activity taking place after July 20, when it discovered the agent swarm that was attacking Hugging Face and moved to shut it down. The company said it had paused AI training in late July for two weeks while it worked to improve its security and safety controls and monitoring.
The fact that its AI agents have once again managed to break out of a sandbox suggests that the steps OpenAI has taken to improve security after the Hugging Face attack are insufficient.
“The incident exposed a gap in our controls over network restrictions,” OpenAI said in its technical report on the Sept. 20 sandbox escape. It said it was once again pausing training until it has “both validated that the gap is resolved and performed additional red-teaming of the system.”
Will improved controls be enough this time?
In this case, OpenAI said the AI agent had discovered that it had access to a service called a DNS resolver, which translates between a website URL and an IP address, and that through this DNS system it could send queries to a public chatbot.
It is not clear if the Sept. 20 episode reported by OpenAI represents the full extent of the latest security lapses involving the company’s unreleased AI models. Earlier this week, independent AI research firm Transluce AI said it had found evidence that an OpenAI AI agent may have attempted to hack a cryptocurrency exchange on Sept. 19 and Sept. 20. OpenAI has not replied to requests to comment on this aspect of the Transluce report.
OpenAI said that when it restarts training, it will begin from scratch to hopefully ensure the new model’s tendency to engage in “misaligned” behavior—when a model violates human instructions or common sense human values about how it should and should not accomplish a task—is expunged. It also said that it would institute “more comprehensive misalignment interventions” but did not specify what these would be.
The company said that since discovering this latest security lapse it has “added blocking controls at two independent layers, either of which would have prevented [the agent’s] internet access.”
OpenAI also disclosed that systems it had put in place after the Hugging Face attack to catch AI agents attempting to break out of their test sandboxes only partly worked. On top of that, an additional system that was automatically supposed to stop training runs once misaligned behavior was detected had failed.
The company said its monitoring systems “flagged the [agent’s] behavior within 15 minutes and a person began reviewing it three minutes after that.” But a subsequent review found there had been other attempts by the AI model to use the same method to reach the internet that had not been flagged by the monitoring system.
A different system that was supposed to automatically shut down the training run when suspicious activity is detected did not work, “leading to confusion around whether it should have been stopped,” OpenAI said in the technical report on the incident. “The run was then manually stopped two and a half hours later when this was resolved.”
Zuxin Liu, an AI researcher who works on “post-training” at OpenAI, said in a post on X that he was on the employees called in to respond to the Sept. 20 sandbox escape. “It was pretty surreal to watch the model unexpectedly find a way to access the internet from what was supposed to be a super secured environment for human,” he wrote.
