Cyber Security

A series of recent reports about cybersecurity vulnerabilities discovered by next-generation AI systems (including Anthropic's announced Mythos model) offer a good illustration of the risks that rapid advances in AI capabilities can pose to society, and the direction that cyber security teams in organisations are headed.

The pattern itself is not new. When OpenAI published GPT-2 in 2019, the debate about whether it was "too dangerous to release" marked an early turning point: the moment AI capabilities became powerful enough that their release started to raise questions around security thinking.

Back in 2019 the concerns were if releasing the GPT-2 model without proper guardrails, might not enable malevolent attackers to create chat systems that could pass for humans in a world where people had started to increasingly worry about bots manipulating online sentiment and elections. ^[6]

What has changed since then is that the focus has become more technical. Research from Google DeepMind, initiatives like the DARPA AI Cyber Challenge ^[5], alongside early previews from Anthropic’s Mythos model ^[1] suggest that next-generation are capable of discovering thousands of previously unknown vulnerabilities in operating systems, browsers, and software that also underpin the day-to-day infrastructure our society relies on to function: power grids, financial systems, government networks, and national defense systems.

Some of these flaws have gone undetected for decades. Not because they were unimportant, but because many of these vulnerabilities are obscure and highly technical, multi-step exploits buried deep in millions of lines of code.

One of the many examples (that serves as a good example of how technical some of these cases are) is a 27-year-old vulnerability in OpenBSD that was recently found by Anthropic's new Mythos model ^{[1] [2]}. The issue involved a detailed bug in the way OpenBSD handles a sequence of data packet manipulations. Which could allow an attacker to crash the system remotely without authentication. OpenBSD is an operating system widely used in firewalls, routing infrastructure, and other security-critical environments with a reputation of being one of the most security-hardened operating systems.

Another example is a new vulnerability found in the 16-year old FFmpeg codec, that is widely used to play and process videos in webbrowsers, streaming platforms, and media applications. The flaw allows an attacker to construct a malicious video file that causes the decoder to write data outside its allocated memory bounds, potentially crashing the application.

Historically, finding these vulnerabilities required such exhaustive technical analysis that it would take highly skilled researchers week to find something like this. Because they were both difficult to find and difficult to exploit, their complexity provided a natural barrier against most attackers.

But that barrier largely disappears for AI agents. An AI system can enumerate thousands of attack paths continuously, iterating in a fully automated way until it finds a way in.

The most effective defense against this is to fight fire with fire. By counter-deploying the most advanced AI systems in highly controlled and secured environments, and making them continuously probe critical infrastructure, we can detect these security vulnerabilities before adversaries do.

To make sure we detect these security vulnerabilities in organisation before attackers exploit them, we need to make sure that security researchers at these organisations have early access to the AI tools that become commercially available to potential adversaries several months in the future. If the most advanced models are available on the open market already, while defensive infrastructure still runs on older generations, the technology gap itself becomes a strategic vulnerability.

Encouragingly, some of this groundwork is already being laid. Anthropic has indicated that its new Mythos model will be made available to a select group of organizations including AWS, Apple, Broadcom, Cisco, CrowdStrike, Google, JPMorgan Chase, the Linux Foundation, Microsoft, NVIDIA, and Palo Alto Networks to strengthen the security of their critical software, before their new model will be made available to the public.

This kind of structured, high-trust access is a promising model for how frontier AI capabilities can be released responsibly, rather than calling for development to halt, it ensures the most capable tools reach defenders first.

Under the EU AI Act, the most advanced general-purpose AI models are classified as posing “systemic risk” if their training exceeded an arbitrary compute thresholds of 10^25 FLOPS. The development of these models is then subject to additional obligations, including mandatory risk assessments, adversarial testing, cybersecurity safeguards, and incident reporting.

While the Act does reflects a growing recognition that frontier AI systems require structured oversight proportional to their potential impact, what is still missing is a formal mechanism to ensure that defenders systematically receive access ahead of adversaries.

Ensuring this might be worth active investment at a national level, with governments funding AI security teams, AI security research, and new regulations to ensure that the most capable commercial models are always made available to critical organisations for early security scans.

This challenge extends beyond a handful of technology partners. Every organization that depends on software infrastructure (which is now essentially every organization) would be wise to set up the internal capacity for AI-driven security research: deploying AI agents for continuous monitoring, running simulated attacks, tracking discovered vulnerabilities, and patching them systematically before they can be exploited. ^[3]

With every organisation needing to set up its own AI agents to counter tomorrows attacks from AI agents this might no longer be a task that can be outsourced to vendors, or addressed with periodic audits.

In the near future, an AI Security Research Team may be as foundational as its traditional security team.