AI agents with legitimate credentials are increasingly making hundreds or thousands of consequential decisions before anyone notices, something that security teams have not really planned for. That problem was at the center of a Black Hat AI Summit panel this week, where Matthew Martin, chief information security and privacy officer at Western Carolina University, Ron Keesing, former chief AI officer at Leidos and now a consultant at Keesing LLC, and Forrester principal analyst Jess Burn explained how autonomous agents are changing security work.
The panel kept returning to an uncomfortable point that while some agentic AI risks are new, most of the issues are security problems that companies never fully fixed.
“I was really adamant that all this is doing is exposing all the bad security we’ve had for a very long time,” Martin said. “What really started changing my mind about this was the whole Hugging Face incident.”
In the past few weeks, new revelations in the OpenAI Hugging Face security breach have shown that agents are able to take advantage of system weaknesses to move at speeds that humans are unable to properly address. In instances where these agents are acting on behalf of security professionals, rather than malicious invaders, this is actually troublesome as agents are now escaping the bounds of their constraints and potentially causing more harm than good.
That leaves defenders with a harder question than whether an agent had permission to act. They have to decide when a chain of legitimate looking actions becomes dangerous.
“One of the key questions for me now is trying to figure out, for example, how do we detect good agent behavior versus bad agent behavior?” Martin said. “Being able to do that in an automated way is fundamentally different from what we’ve done before.”
Identity Security Was Designed Around People
Corporate identity systems were built for human users. A person can be fired, prosecuted or asked why they accessed a sensitive file. An AI agent has no comparable sense of consequence. Keesing said that difference cuts into assumptions embedded deep in identity management.
“We actually want attribution,” he said. “We want people to understand that there are rules and there are consequences for breaking the rules.”
That deterrence model makes sense for employees, but it makes far less sense for software and autonomous agents acting on their own behalf.
“Agents don’t face that deterrence factor that’s built into a lot of reasons why we architect the way we do,” Keesing said.
Speed presents another problem. Most identity monitoring systems assume a pace of human behavior. An employee might generate a limited number of access decisions that matter during a workday. An agent may generate hundreds or thousands. It can create other agents and multiply that activity again.
“We’ve built identity management practices, identity monitoring practices, around the speed with which human beings act,” Keesing said. “The time scales are completely different when you’re talking about machines.”
Martin pushed the idea further. Companies may eventually need something resembling a personnel function for software workers.
“There’s got to be like an HR for AI agents where we can sort of manage them in the similar way that we do humans,” he said.
That could mean assigning every agent a role, an owner, a defined set of permissions and a clear point at which those permissions expire. The concept sounds unusual only if companies keep thinking of agents as software features rather than actors performing work.
AI Security Vendors Face An Increasingly Difficult Sales Pitch
AI security startups are multiplying, but Martin questioned whether CISOs need another console and whether the vendors can properly restrain and control them unless in corporate systems.
The panel made the point that buying agent monitoring software does little good if its alerts never connect with the identity, network and application signals the security operations center already watches.
Martin said the calculation becomes especially difficult for security leaders with small teams and tight budgets.
“It’s not just, is there a tool and do I have a budget to buy that tool, but then do I have the budget resources to actually do anything with it?” he said. “I think the other side of that that’s really important is even if you can afford some of these cool new tools for agentic monitoring, just deploying those tools is not going to be enough for you either.”
One answer may be architectural rather than product based.
Keesing argued that organizations should think carefully before allowing agents to interact directly with core systems of record. Instead, they may need a separate data layer above those systems where agents can operate with less chance of damaging or exposing the original records.
“That data layer is a critical point of insulation,” he said.
The implication is difficult for vendors hoping to sell AI security as a new standalone category. Much of the hard work may still sit in old disciplines such as identity, data architecture, application security and basic operational discipline.
The skills problem is posing just as much of a challenge as the technology.
“The knowledge right now that’s required to secure agentic AI is changing faster than most organizations can write job descriptions correctly,” Burn said.
Martin does not think smaller organizations should chase narrow credentials for technologies that may look different six months from now. He would rather hire people who are curious enough to keep learning.
“If I can find somebody who is super passionate about it and willing to put in the time to research and learn and go play around with it, that’s the people I want on my team,” he said.
Large companies have more room to specialize. Martin expects some of them to build AI security groups resembling application security teams, staffed with people who understand AI but still know the basics of access management, APIs, data flows and secure engineering.
More critically, technical fluency is only part of the job, especially when the state of the art for agentic systems continues to evolve. Someone has to recognize when the machine is wrong.
“I think judgment has been a word that we’ve been talking about a lot right now,” Martin said. “If you don’t know whether something is correct or not, poor decisions could be made.”
Keesing sees another shift coming inside security teams, with more time spent attacking their own systems before somebody else does.
“We’re an organization that’s just continuously red teaming everything we do and every public facing aspect and attacking ourselves and then remediating those vulnerabilities before anyone attacks us,” he said.
AI has lowered the skill barrier for attackers. Martin noted that flaws once written off as difficult to exploit may no longer deserve that comfort.
“Things that we could have previously said, not that big a deal, it’d be really wicked, it would take somebody very talented to take care of that, and we’re not a big target, so probably okay, versus now that’s not the case,” he said.
