We’ve officially entered a new era of cyberattacks. November reports from Anthropic and Oligo Security detail the use of jail broken LLMs to carry out large scale cyberattacks. In both cases, the companies claim that LLM based code generation and, in the Anthropic case, other LLM capabilities, were used in executing attack campaigns.
This should not be a surprise. Researchers at Cornell University predicted we were on this path in May of this year. The reality is that LLMs are incredibly useful tools for a wide variety of tasks, some of which happen to include those that are relevant to cybersecurity. OpenAI has even released a dedicated cyber security researcher. But human history includes many examples of technologies not originally developed for war but were later weaponized. Perhaps the most famous example is dynamite, which ironically was invented by the namesake of the Nobel Peace Prize, Many, many other examples exist: fertilizer, commercial airliners, 3D printers, drones…the list goes on and on.
The Implication For Cybersecurity
In the cybersecurity context, it means that LLMs have been turned into attack tools by cybercriminals and nation state threat actors. The long term implication of this is that the approaches that worked in the prior era are no longer going to work in the era of AI-generated or even just AI-assisted cyberattacks.
In the prior era, attackers had to choose between going deep (high-value targets, high effort) or going broad (scripted spray-and-pray attacks). Generative AI collapses that tradeoff. With well-crafted prompts, an attacker can now do both: create human-level attack campaigns and apply them to a large number of targets simultaneously without human intervention or ongoing direction.
In the Anthropic case, the LLMs were given initial direction on targets and attack frameworks by human operators, including an approach to jail-breaking the underlying LLM (Claude code in this case) to circumvent the built in guardrails against malicious activity. From there the execution of the campaigns was largely autonomous and resulted in attacks on roughly thirty targets and a small number of successful breaches, according to Anthropic’s report.
In the Oligo-reported case, researchers describe a botnet that uses AI-generated code to attack, Ray, an AI infrastructure software tool and then go on to mine cryptocurrency and autonomously identify other systems to attack.
Until today, security vendors and practitioners have largely responded to the challenges of vulnerability management by prioritizing fixes according to a threat model. Traditionally, threat modeling combines the value of an asset with the severity of the potential exploit and the likelihood of a potential exploit. It’s not perfect nor is it meant to be…it’s meant to focus security efforts in the nearly ubiquitous scenario in which an organization doesn’t have the resources to address every possible threat.
The risk-based prioritization approach mainly works because the cost of an effective attack is too high for the attackers to explore every possible avenue of attack. It makes the implicit assumption that human attackers will focus where there is the most value to be gained and the most likelihood of success. It’s a rare case in which the attacker/defender dynamic favors the defender. Though it should be noted that there’s still a fundamental asymmetry that favors the attackers. Defenders in theory must get to every available attack path while attackers only have to find those the defender missed. When you can assume attackers have to be choosy about their own use of resources, as a defender you can make good guesses about where they will focus.
But when attackers can collapse the depth and scale tradeoff by using LLMs as a productivity tool, they don’t need to focus on the highest value assets and/or most likely to succeed attack paths. This takes away that small defender advantage and makes the dynamic even more asymmetric.
Another complication of defense is that when considering closing vulnerabilities, the defender doesn’t only have to worry about how their actions affect potential attackers. Way more important is how mitigating actions might affect legitimate use. If a mitigation closes out an attack path but also shuts down a key function then the damage has been down without the need for an attacker at all. In many cases, mitigations are left undone for precisely this reason, making the job of the defender even more difficult.
Ironically, AI Can Help Solve This Problem
What’s required to address this new reality is a shift in mindset. Rather than a prioritized backlog of security fixes coordinated by a ticketing workflow system but ultimately performed by human operators, organizations will need to harness AI to address the threats. This means an automation first mindset that doesn’t ignore the operational risk of making changes that can disrupt the business. This doesn’t mean prioritization goes away. It means that operational risk should be factored in when designing fixes and it also means that rather than a binary fix/don’t fix model, additional context will be needed in designing mitigation strategies.
For example, if there is a serious active threat to a specific business application, the answer can’t be to disallow all use of that application across an organization. A better solution would be to disallow it for all users that have never used the application, or perhaps haven’t used it recently, but to allow it, perhaps with some added controls, for those users that do need it for their role. There’s a lot of potential nuance in that approach. Typically, a human security engineer would make that decision, but in this new era, AI can suggest an approach and also evaluate the operational risk of implementing the proposal. If the risk is low enough, maybe it gets automatically rolled out. If the risk is high, a human security engineer might be need to make the call.
We’re still facing an uphill battle against potential attackers and their ability to use AI to scale their efforts makes that disadvantage worse. But that same technology can be used to level the playing field. With the right approaches we might even be able finally to tilt it in our direction. As a disclaimer, I invest in primarily cybersecurity companies and I’m actively evaluating companies taking such an approach.
