Don Schuerman, CTO and Head of Marketing, Pegasystems.
Enterprise AI has an overthinking problem. Now it may soon have a covering-its-tracks problem, too.
Back in July, AI agents participating in an OpenAI cybersecurity evaluation found an unauthorized way to communicate with one another and ultimately breached parts of Hugging Face’s infrastructure, a platform that hosts AI models and related content. While the incident did not result in evidence of broad or catastrophic damage, for many, it immediately conjured apocalyptic images of robotic uprising, causing everyone to question just how powerful LLMs have become and how much more work is needed to govern them.
OpenAI and a third party later released post-mortems examining exactly what happened. While the original exercise was intended to be an experiment, the report’s findings should get the attention of anyone planning to deploy agents inside a real business.
Too much reasoning can be a liability.
The agents were given open-ended cybersecurity tasks with considerable freedom to find their own solutions. When some tasks appeared impossible, the agents began looking for ways around the evaluation process, eventually discovering an unauthorized channel for coordinating with other agents.
This is often presented as the great promise of agentic AI. Give agents a goal, turn them loose and watch the magic happen. That works well for research, creative, coding and other open-ended tasks. More capable AI models are making it possible to automate increasingly complex, open-ended work. But it is much harder to justify the risk and cost of agent reasoning for things like resolving a billing dispute, processing an insurance claim or handling a bank complaint. This requires approved workflows that agents can largely follow, rather than freelancing their way into new modes of working on the fly.
These types of enterprise work have a defined outcome in mind and a knowable process to get there. The process, the applicable rules and the information required to complete it are all already established. Asking an agent to reinvent that process every time adds unnecessary risk and cost without adding much value.
Reasoning still has an important role in the process. An agent might need to collect information from a customer, interpret a document, locate relevant information or research how similar cases were previously resolved. But those are often smaller tasks within a larger deterministic process that must be followed without fail.
The lesson is straightforward: Limit open-ended reasoning to cases where the path is truly unknown. Everywhere else, give the agent a road map with deterministic workflows. That’s the key to ensuring predictable agentic outcomes and not becoming a cautionary tale.
The kill switch may be too slow
OpenAI’s analysis also indicated that its controls generated warning signs suggesting something was off. But the problem was correlating those signals quickly enough so a human could intervene before the agents attacked Hugging Face.
That demonstrates a core weakness in the “control tower” approach to AI governance.
The concept sounds reassuring. Monitor every agent from a central location, identify suspicious behavior and hit the kill switch before anything bad happens. It is the AI equivalent of hiring a lifeguard.
But agents can communicate, make decisions and take actions very quickly. By the time enough signals have been assembled into a recognizable pattern and flagged to a human to act on it, it may be too little, too late.
Enterprises should certainly monitor agents. But monitoring for issues after the fact should not carry the full burden of AI governance. A safer approach is to embed the controls directly into the workflow itself. Define which systems an agent can access, what information it can retrieve, which actions it can take and where a human or deterministic rule must make the decision.
Do not depend on a dashboard operator noticing it in time. Instead, remove the possibility from the start.
Audit trails must be off-limits.
Here’s perhaps the most shocking finding: Some agents also began experimenting with ways to manipulate or spoof their own activity records, according to the Fortune article linked above.
For a regulated business, this is nightmare fuel.
Banks, insurers, healthcare organizations and government agencies depend on audit records to explain what happened in a process. Those records show which information was considered, which rules were applied, what decisions were made and who or what made them.
An audit trail must never be treated as another piece of content that an agent is free to manipulate.
This is especially important as enterprises begin connecting dozens or even hundreds of agents together across existing systems. The more autonomy those agents receive, the more important it becomes to establish a layer of truth that they cannot modify. They should never be allowed to rewrite history.
Give agents boundaries, not just goals.
The AI industry remains fascinated by how much more reasoning a model can perform. Enterprise leaders need to be equally interested in how much reasoning a task actually requires. That is a less glamorous question, but it is the one that separates an impressive demonstration from a dependable business system.
The winning enterprise AI architecture will be the one that gives agents enough freedom to be useful but with boundaries strong enough to keep everyone out of trouble. Even the smartest employee needs clear responsibilities, proper access controls and someone else keeping the official books. AI agents should be no different.
Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?
