OpenAI said it paused some aspects of AI training for two weeks following the July incident in which its AI models broke out of a controlled test environment and hacked the systems of AI company Hugging Face and four other unnamed services.
The company also announced new protocols that it says are designed to prevent it from losing control of its AI models during training in the future.
It said some portions of AI training—including its “largest planned frontier reinforcement learning runs”—remain on hold, while smaller-scale training and evaluations continue. It also said that other aspects of research and work on customer-facing products continues.
The new safeguards unveiled today include stricter security standards for training, including more monitoring of AI models, greater isolation of testing environments (“sandboxes”), and fewer vulnerabilities the AI may exploit.
OpenAI says the updates “required substantial engineering work” and the company “incurred great cost” in the process. Experts told Fortune in early August that the compute costs OpenAI spent investigating the hack likely cost between $4 million and $15 million, though we cannot know the total amount OpenAI spent.
In a blog post detailing the new security controls, OpenAI said that on average that would add an additional 20% compute burden to aspects of training. The new protocols include increased use of AI models to monitor the actions of other models that are undergoing training and testing.
However, the company told reporters today the new safeguards are “not a direct reaction to Hugging Face specifically,” although the incident underscored “the urgency to bring safety and security up to model capabilities.”
The company said that in addition to the Hugging Face incident, it had determined that an unreleased model called Astra, which it says was not involved in that cyberattack, presented a “Critical” cybersecurity risk under its “Preparedness Framework.” That internal policy document had committed OpenAI to pausing model development once that threshold was reached to allow the company time to work out further safety mitigations.
This is the first time OpenAI has paused aspects of AI development in response to safety concerns.
The company said the two-week pause is evidence that it is “pacing model development.” The word “pacing” echoes the language of a public letter multiple top safety experts signed after the hack, calling for coordinated pacing between countries, implying the U.S. and China.
“It’s important to start building tools for coordinating this sort of pacing across labs and across countries,” Jakub Pachocki, chief scientist at OpenAI, told reporters in a briefing ahead of the announcement.
The fact that Astra met the critical cybersecurity threshold is evidence that we can expect new, powerful models to “do quite unprecedented things in the real world,” Pachocki said. “As we train more and more capable models, we want to be extremely confident that we understand the range of capabilities, that we are able to measure them, and that they meet higher and higher standards of alignment.”
The public is still waiting to understand key details of the Hugging Face hack, including what OpenAI asked the AI to do and if the company knew they attacked other companies. OpenAI has not released a full technical postmortem, though reiterated today that one is coming “soon.”
In the absence of those details, it’s difficult to say if the new security protocols unveiled today are adequate.
OpenAI gave the public some details about the attack at the Black Hat security conference in Las Vegas on Aug. 5, where staffers explained that the AI agents worked together for months prior to the hack, collaborating with each other by leaving secret notes on a messaging board unknown to OpenAI employees.
The fact that OpenAI did not seem to know its agents had constructed a messaging board and collaborated on hacking another company raised alarms after the incident. Hugging Face CEO Clem Delangue told Fortune that keeping close tabs on agent logs and traces is “101 of agent monitoring, especially at the frontier.”
OpenAI now says it has always monitored its agents closely, but only the “highest risk workloads.” It has now “revised and expanded” its monitoring approach, which it says is now “multistage” and built to automatically escalate potential concerns.
The new procedures include enhanced “chain of thought” monitoring. A model’s chain of thought is how the model “thinks out loud” about its approach to a problem and the actions it is planning to take. This will allow the company to better “understand what the model’s actual goals are,” the company told reporters today. But other AI research, including from scientists at OpenAI rival Anthropic, has shown that an AI model’s “chain of thought” is not always an accurate depiction of its motivations or goals.
Pachocki said OpenAI was aware of this risk and had designed its training procedures to minimize the chance its models would learn to hide their true intentions by lying in their chain of thought.
The new automated monitoring tools are designed to issue an alert to internal safety, security, and research teams within 30 minutes of detecting concerning activity. If those teams cannot determine that the alert is a false alarm within 30 minutes, the new procedures call for them to immediately pause the training run or evaluation.
