When Silicon Valley and the Holy See agree, it is worth asking what they know that governments do not.
On Monday, Pope Leo XIV published his first encyclical, Magnifica Humanitas, calling for AI to be disarmed and regulated in the service of humanity. Standing beside him at the Vatican was Christopher Olah, co-founder of Anthropic, who acknowledged that AI companies operate “inside a set of incentives and constraints that can sometimes conflict with doing the right thing.” Separately, Anthropic’s CEO Dario Amodei has stated that “the next tier of risk is actually AI companies themselves” — and that AI leaders, including himself, should not be the ones deciding the technology’s future.
This is the architects of the most capable AI systems in existence telling the world they cannot govern themselves alone. The question is no longer whether global AI governance is necessary. It is whether it will be designed before a crisis makes the answer obvious to everyone.
The industry knows — and says so
Inside the industry, few serious people dispute the core risks: labour market disruption, autonomous weapons, child safety, mass surveillance, the erosion of human oversight, and the prospect of systems capable of recursive self-improvement. Anthropic has built Constitutional AI and a Responsible Scaling Policy. OpenAI maintains a Preparedness Framework with structured red-teaming. These are serious efforts. They share one structural limitation: they are internal. Each company sets its own thresholds and its own level of transparency.
Internal governance cannot resolve an external problem. The companies building frontier AI operate in a competitive environment where unilateral restraint is a strategic liability. A company that slows down when a rival does not has not made the world safer. It has simply lost ground. The same is true for states: no government can credibly restrain its own companies if China, or any other major AI power, is not bound by a comparable commitment. The result is a textbook collective action failure. Everyone understands the risk. Nobody has the individual incentive to absorb the full cost of solving it. That is not a market problem. It is a governance architecture problem.
The right analogy is not nuclear — it is finance
The comparison most often reached for is nuclear arms control. It is the wrong analogy. Nuclear weapons were state programmes with identifiable facilities and verifiable tests. Advanced AI is developed by private actors on commercial infrastructure already embedded globally.
The financial system offers a more useful model. Banks compete fiercely across jurisdictions, connected in real time, with systemic risk from terrorism financing and financial crime that no single institution can contain. The response, built around the G20’s FATF standard-setting framework, combined shared norms, operational standards, governmental and private-sector obligations, expert review, and real consequences for non-compliance. While it is far from perfect, it did change behaviour at scale, without requiring a decades-long treaty ratification process, because it distributed accountability across a coalition wide enough to make evasion costly, dividing safety obligations across both governments and private sector.
I spent years designing and evaluating those standards, including leading international coordination against ISIS financing and shaping the crypto framework now adopted by more than 120 countries. I have evaluated national compliance programmes directly. I believe I learned what makes an accountability mechanism work — and what makes it fail.
What governance could look like
A workable AI governance architecture starts not with technical specifications but with high-level norms: shared commitments on risks that already command genuine consensus. Those norms translate into criteria that assess real-world outcomes, rather than prescribing technical processes that will be obsolete within months. Independent expert panels, composed of representatives from both the public and private sectors, assess whether companies and states are meeting those criteria. And the mechanism needs consequences: compliance unlocks trust, market access, and cooperation; non-compliance carries escalating costs. A bilateral U.S.-China understanding would be essential, but not sufficient; only a broader coalition, such as the G20, can turn restraint into durable global pressure.
The window is not unlimited
Some credible estimates place Artificial General Intelligence — systems more capable than humans across most domains — as early as 2030. When machines surpass human capability, humanity’s only adequate response is a collective one. No single company, and no single state, can manage that alone.
Amodei has called for candour from AI leaders and governments about how fast this is moving. The Pope has called for the technology to be disarmed before it disarms us. Olah stood at the Vatican and confirmed the incentive problem his own industry faces. The convergence from opposite ends of human authority — moral and commercial — is itself a data point policymakers cannot dismiss.
The international community has united around existential shared risks before — nuclear proliferation, terrorist financing, systemic financial collapse. It built coordination mechanisms that held, imperfectly but consequentially. AI demands the same, and more urgently. The risks to humanity are of a different order of magnitude. The catastrophes are not hypothetical. Waiting to govern after they occur is not remediation. It is surrender.
Governments and international organizations now need to convene that process. Bring in the people who built these systems and understand their failure modes from the inside, as well as experts with hard-won experience coordinating between parties that do not trust each other — be it countries or commercial competitors — and translating shared risk into binding commitment. The window is closing. We owe our children governance built before the catastrophe, not as a response to it.
The opinions expressed in Fortune.com commentary pieces are solely the views of their authors and do not necessarily reflect the opinions and beliefs of Fortune.
