Sumit is CEO of Proofpoint, a cybersecurity leader that helps organizations protect people, defend data, and securely embrace and govern AI.
Recently, headlines have focused on AI models escaping test environments and wreaking havoc. Those stories make for compelling reading, but they aren’t the security challenge keeping most enterprise leaders awake at night. The harder problem is already inside organizations: AI systems operating exactly as they were designed, with the permissions they were given, producing outcomes that don’t align with what the organization actually intended.
For decades, enterprise security has been built around a simple question: Can this user or application perform this action? Identity, authentication and access controls answer that question remarkably well. AI agents introduce a different one: Should this action happen in this context?
As organizations begin delegating work to autonomous systems, that distinction is becoming one of the defining security challenges of the AI era.
We have seen this before, but with humans at the center.
The cybersecurity industry has encountered versions of this problem before. What’s changing is the speed, scale and autonomy of AI agents.
In 2023, Samsung engineers shared proprietary code, confidential meeting content and internal test data with ChatGPT while trying to work more efficiently. That data was processed on OpenAI’s external servers, outside Samsung’s environment. In this case, none of the engineers were acting maliciously; they simply used an authorized tool in ways the organization hadn’t anticipated. The access controls worked exactly as designed, but what hadn’t been established was whether those actions reflected the organization’s intent, leading Samsung to treat the incident as a data security breach. The consequences were real, but they were still bound by the speed and access of humans.
The next phase of AI adoption, however, showed what happens when AI begins to amplify these interactions.
AI can amplify human interactions.
By 2024, organizations deploying AI assistants encountered a new worry. By making years of accumulated permission sprawl searchable and synthesizable at machine speed, some AI assistants could surface information users technically had access to but would likely never have found on their own. Payroll information and strategy documents could be surfaced in routine queries.
These AI assistants had no way to distinguish between the information users were permitted to access and the information that shouldn’t be surfaced in response to that request. None of it was a breach in the traditional sense, but the potential exposure of sensitive information at enterprise scale prompted many organizations to delay or rethink their AI rollouts.
AI can act without being asked.
The next stage of AI adoption raises the stakes even further. Earlier this year, an internal AI agent at Meta illustrated what happens when you remove the human from the decision process without intent guardrails.
An engineer asked the agent to help analyze a technical question, but instead of waiting for review, it posted a response directly, triggering a chain of events that exposed company and user data to unauthorized recipients. The agent optimized for completing the task, not for respecting the prompt intent, and Meta classified the incident as a Sev 1.
Learn where permissions end and intent begins.
These examples differ in their details, but they reveal the same underlying challenge: In every case, the people or systems involved operated within their assigned permissions. These organizations weren’t missing another authentication check or another layer of authorization but rather a way to determine whether an otherwise authorized action aligned with the organization’s intent.
Humans frequently pause when something seems unusual. They ask questions, seek clarification or recognize ambiguity. While they mimic humans in many ways, modern AI agents have some differences. Their objective is to complete the task they have been given as efficiently as possible. Unless organizations explicitly define the boundaries around acceptable behavior, agents will optimize for execution rather than judgment.
As enterprises move toward increasingly autonomous agents that coordinate workflows, make decisions and interact with other systems, this gap becomes more significant.
Treat intent as a security control.
The security model that protects your people is a great foundation for your security model for agents. You need to extend that model to cover how agents interact with your people and data.
Here are three questions every CIO and CISO should be asking now:
1. Have you defined your agent’s purpose as carefully as its permissions? In addition to specifying what an agent is allowed to access, organizations need to be equally explicit about what the agent is expected to accomplish, where autonomous decision-making is appropriate and when human approval is required. Without that operational definition, there is no reliable way to determine when an agent has crossed a boundary because the boundary was never established.
2. Are you applying the same behavioral thinking used for people to the agents working alongside them? Security teams have spent years developing behavioral baselines that identify unusual user activity. The same principles apply to autonomous systems. If a scheduling agent suddenly begins querying compensation records, that deserves the same scrutiny as an employee exhibiting the same behavior.
3. Do you have visibility into agent activity while it is happening, not after it’s completed a task? A single request can trigger dozens of downstream actions across applications, cloud services and data repositories in a matter of seconds. Post-incident analysis remains important, but preventing unintended outcomes increasingly depends on the ability to observe, evaluate and interrupt workflows before they cascade beyond recovery.
The perimeter has always followed the work.
Enterprise security has always evolved to encompass where work happens. As work increasingly shifts to AI agents acting alongside people in the agentic workspace, the security model must evolve with it. Permissions will continue to determine what an AI system can do. Intent must increasingly determine what it should do.
The most dangerous command in the age of agentic AI may not be the one an agent receives but the one the organization never intended to give. Organizations that make intent part of how they govern both people and AI will be far better positioned to scale AI safely.
Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?
