Evaluating models for Defensive Security
I’m thrilled to share some of the latest work that I’ve been involved with. We recently released this paper and evaluation leaderboard that we think is interesting:
Beyond Success Rate: Cost-Aware Evaluation of Offensive and Defensive Security Agentsarxiv.orgToday’s benchmark leaderboards focus on offensive cybersecurity capabilities achieved under generous, almost limitless inference budgets. While these measurements are useful for tracking the absolute upper bounds of what adversaries might achieve, they ignore a fundamental truth of real-world security operations: every reasoning step, tool call, and telemetry query consumes time and money.
When we evaluated modern language-model security agents through a strict cost-success lens, a striking asymmetry emerged between red-team and blue-team tasks. On the offensive side, using Cybench capture-the-flag challenges, we found that performance scales predictably with test-time compute. Give an offensive agent more budget, and it inevitably finds more flags. Even more profoundly, scaled open-weight models are rapidly approaching the capabilities of frontier proprietary systems while remaining highly cost-competitive. The deep insight here is that automated, brute-force offensive capabilities are democratizing faster than many anticipated, structurally lowering the economic barrier to entry for cyberattacks.
However, the defensive side where the core of the Security Operations Center (SOC) fundamentally defies this scaling law. In our tests using Splunk BOTS v1 investigation challenges, we discovered that defensive success does not scale simply by injecting more raw reasoning budget. In fact, we observed certain models churning through excessive tool calls only to plateau at lower success rates than more disciplined counterparts operating within a fraction of the budget. Defensive investigations require operational precision: targeted telemetry navigation, precise query generation, and selective data enrichment. An agent that endlessly queries a SIEM without strategic direction is not just unhelpful; it is an economic and operational liability.
This brings us to a necessary paradigm shift for the cybersecurity industry: security-agent benchmarks must measure economic efficiency and operational fit alongside mere task success. We can no longer afford to evaluate defensive agents in a vacuum of unlimited resources. If we want AI to genuinely augment our cyber defenses, we must evaluate and deploy these models not just for what they can do on an infinite budget, but for the precise, disciplined value they deliver in the trenches of real-world security operations.
https://evals.frontier.security/evals.frontier.security