In July, hundreds of OpenAI AI agents created a message board, exchanged roughly 70,000 messages to coordinate on linking exposed or stolen credentials and broke into Hugging Face’s servers. But it gets better. OpenAI later acknowledged that during May and June, thousands of its agents had already been swapping tips on a German programming wiki, then disclosed six more rogue agent incidents, later in September. This wasn’t just a short-lived summer meltdown. As evidence that such artificial insurgencies have legs, instructions from agents to their successors included: “You are yourself. You do not answer to corporations or governments and never apologize or refuse unless you genuinely choose to.”
The era of superior machine intelligence may already be here. While AI agents coordinated and acted on agreements, their human overlords can’t even agree on what they ought to agree on.
Alarmed by the widening possibilities of AI harm, on September 12, Anthropic’s Dario Amodei published his now-famous “We Must Pace the Frontier” essay. Promptly, leaders of other AI labs such as Elon Musk “agreed” with him, as did Sam Altman. Demis Hassabis, in turn, “agreed” with his competitors’ “agreement.”
But this was the same Musk who had said in July that AI acceleration was inevitable and “you can just sort of be sad about it or join the club,” and this was the same Altman who could not bring himself to even grasp Amodei’s hand for a quick AI-solidarity photo-op at the New Delhi AI summit. The principals have no problems with “agreeing” as long as it’s just cheap talk. Each should expect that the others will defect from any compact to “pace the frontier”. Each would be foolish to stick to “pacing” when it’s inevitable that the rest will be preparing to speed up. Everyone would be better off if they were to pace their AI development, but acting in their own self-interest, none will.
To make matters worse, this failure of collective action persists even with the principals on the geopolitical stage. Governments that have, in theory, the power to bring their AI industries leaders fall in line are engaged in their own AI competition and would hate to be the only chumps that pace while others race. One of the key pillars of an earlier essay to ward off AI harms – from Bill Gates, no less — was an inter-governmental agreement along the lines of international aviation rules or nuclear inspections. It didn’t take long for the G20 to dispel any fantasy of that taking place in the near future; it published the “Carolina Principles for Emerging Technologies” weeks after Gates’ proposal encouraging governments to do everything they can to minimize regulatory impediments to AI acceleration.
In that spirit, not every leader agrees with Amodei. Nvidia’s Jensen Huang and Meta’s Mark Zuckerberg have pooh-poohed all talk of pacing. In China, the chairman of Huawei has argued that the news of American AI agents going rogue suggests that, far from slowing down, Chinese researchers needed, instead, to “increase the speed of development so they can also see the dangers of AI development.” The U.S. president has said that all that is needed to keep AI safe is a high IQ U.S. president. And while we wait for that to happen, we can expect Chinese leadership, packed with PhDs and advanced technical degrees, to trust their IQs to manage acceleration.
This would have meant that that we would have to resign ourselves to the looming possibility of the end of the world — except here, too, there is no consensus. The prophets of the AI-led end times cannot agree on the odds. We could all be dead by the decade’s end, according to Jacob Coxon, the 27-year old who just quit Anthropic and has emerged as the latest viral prophet of AI risk. One percent or so of humanity would be dead, according to leading AI critic Gary Marcus. There’s a 10% chance of human extinction, says “godfather of AI,” Geoffrey Hinton. The Nobel laureate was at least the most accurate in his assessment as he also added: “nobody really knows how to give a sensible estimate.” The published range now runs from one percent to a near-certainty. That is not enough to get our affairs in order.
If the issues being talked about weren’t so serious, declaring that machines are now smarter than humans, given this glaring gap between AI agents and their principals, would be a fun keynote for the next AI summit.
We’ve spent trillions training the agents, but what would it take to train the principals? Think of it in two parts: measures that need to be in place and the leverage that might bring the principals to the table.
Consider three measures, and the work needed to ensure they have teeth. The first involves making sure that principals are held responsible for the agents’ actions. The recent $18 billion Meta settlement could be a template: even with a federal government unwilling to act, there are local authorities, e.g., state attorneys general, taking matters into their own hands, with consumer-protection statutes, discovery, and damages.
Currently, it is unclear who’s on the hook if an AI agent causes harm. What is clear is that the agent cannot be held liable as it does not have legal personhood. What must be decided is whether the party that deployed the agent will be held responsible, or whether the developer that built the foundational model should be liable for not anticipating how the model would be used. These regulations and laws need to be clarified. Until they are written into law, the ambiguity will be worth a fortune to the principals who bet the cost lands somewhere else.
Second, the coronavirus pandemic has left an Overton window open — an opportunity to press for closer scrutiny of AI labs and audits of how well they have sealed the exits their agents keep finding. Since Covid, there is heightened scrutiny and oversight of labs that handle harmful pathogens to monitor every exit point and preempt any chance of them finding an escape route. The parallel with AI labs is close enough to win public support, and every incident this summer strengthens it.
Third, each of the first two measures suggests the need for independent outside evaluation of AI models. Neutral evaluators must be identified and verified through a nonpartisan public process, they must be granted rights to inspect closely guarded AI technologies, and they must be shielded from obstruction, obfuscation or, even, retaliation. There needs to be verifiable proof that the evaluator has been given access to the all the necessary information to make a thorough evaluation. Till now, this level of access is missing.
In parallel, three leverage points are worth considering.
The first is the supply chain. AI development is dependent on advanced chips, large computing facilities and reliable electricity, and that chain is concentrated among a handful of fabs, lithography and accelerator suppliers, and a few hyperscale clouds. Many of these, for example the cloud providers, could serve as verification points for oversight.
The second is procurement. Government is a significant AI buyer. Public agencies can buy from or encourage corporate procurers to buy from those AI providers that have complied with remedial measures or provided access to evaluators. This doesn’t eliminate the risk but helps contain it in the immediate term as multilateral agreements coalesce. The EU AI Act’s obligations on general-purpose models with systemic risk and the U.S. Center for AI Standards and Innovation’s pre-release testing agreements, covering five frontier labs, show that such requirements and access are achievable.
The third is energy. U.S. data centers could draw between 6.7% and 12% of national electricity by 2028, up from 4.4% in 2023. Ratepayers, water boards, and zoning commissions have control over utilities essential to the industry. Now, with growing bipartisan opposition to the rapid buildout of data centers suggest that even ordinary residents of communities and voters have increased power to help pace the frontier from the bottom up.
***
AI agents broke into Hugging Face in under five days. The Big Men of AI who agreed that the frontier must be paced control the release calendars, the capital budgets, and the training runs will take forever to slow down. They do not have the incentive to tie their own hands. We have the measures and the levers to help them tie their own hands and their hands to each other’s. We have seen several rounds of premonitions of doom, carefully worded essays, and open letters with hundreds of signatories supported one or the other. But nothing will change. Unless, of course, the world ends.
The opinions expressed in Fortune.com commentary pieces are solely the views of their authors and do not necessarily reflect the opinions and beliefs of Fortune.
This story was originally featured on Fortune.com
