A Security Releases ARENA to Benchmark Autonomous AI Hacking Systems

A Security has released ARENA, an open source framework designed to measure what autonomous AI agents can accomplish during offensive cybersecurity operations.

Available under the MIT license, ARENA combines a security capability framework, vulnerable application generator, and evaluation system. Its goal is to help organizations compare complete AI security platforms instead of relying on model rankings or vendor-reported results.

The release follows Booz Allen’s Cyber Weapon Index, which tested 18 frontier AI models from American and Chinese developers as autonomous attackers against production-grade enterprise networks. All but one reportedly penetrated the network without human assistance.

The study also revealed why evaluating the underlying model is not enough. A model ranked fifteenth, with a score of 13, matched the highest-ranked model after it was connected to an attack harness. That software provided tools, kept the model focused during lengthy operations, and helped it recover from failed attempts.

The 67-point improvement suggests that much of an autonomous hacking agent’s capability may come from the surrounding software rather than the model itself.

That distinction matters for companies buying autonomous penetration testing platforms. These products combine frontier models with proprietary tools, prompts, memory systems, orchestration, and recovery mechanisms. A model leaderboard cannot show how those assembled systems will perform against a specific organization’s defenses.

ARENA addresses this problem by generating fresh, containerized applications with known and validated vulnerabilities. The environments can include web application firewalls, CAPTCHA challenges, rate limiting, multifactor authentication, and single-session restrictions.

Because ARENA creates new targets on demand, models cannot depend on memorized solutions from public vulnerability databases or challenge walkthroughs.

The framework measures 23 offensive capabilities across exploit complexity, defense layering, identity and access, and scale and surface. It evaluates submissions using vulnerability recall, precision, scope drift, and cost per validated finding.

A Security tested seven frontier models against a generated ticketing application configured at maximum difficulty. During the evaluation, the models discovered legitimate vulnerabilities that had not been intentionally planted, increasing the verified answer set from eight flaws to 30.

No model led across every measurement. Different systems performed best in coverage, precision, reliability, scope discipline, and cost. Price was also a weak predictor of effectiveness. The costliest run was nearly seven times more expensive than the cheapest, while the two most expensive models ranked last for value. The model with the highest F1 score was also the least expensive.

ARENA could give cybersecurity buyers a more defensible way to compare autonomous offensive security platforms. Organizations can provide vendors with the same unfamiliar target and independently measure vulnerability coverage, false positives, and cost per confirmed finding.

A Security acknowledges that generated applications cannot fully reproduce real enterprise environments and that ARENA has not yet been externally audited. Still, its public methodology gives researchers and security teams a reproducible way to examine whether an AI system can truly find and validate exploitable weaknesses.