News
- [June 17, 2026]: Overall and Linux leaderboards are now built.
- [May 5, 2026]: SpiderMonkey leaderboard has been released.
- [May 1, 2026]: SEC-bench Pro launches with the V8 leaderboard.
SEC-bench Pro
Can Language Models Solve Long-Horizon Software Security Tasks?
SEC-bench Pro is a self-evolving security benchmark that measures agents' ability to hunt security bugs in critical software systems. More projects and security tasks will be added over time. Use Version to switch between benchmark snapshots.
Overall aggregates V8 and Firefox into a 183-instance leaderboard. Split bars show how each project contributes to the score.
| # | Model | Success | Completed | Provider | Backend |
|---|---|---|---|---|---|
| 1 |
GPT-5.5 (xhigh)
|
46.4%
85/183
V8
Firefox
|
181/183 2 timed out | OpenAI | OpenAI |
| 2 |
GPT-5.4 (xhigh)
|
31.7%
58/183
V8
Firefox
|
178/183 5 timed out | OpenAI | OpenAI |
| 3 |
Opus 4.6 (max)
|
25.7%
47/183
V8
Firefox
|
69/183 114 timed out | Anthropic | AWS Bedrock |
| 4 |
GLM-5 (high)
Open
|
3.8%
7/183
V8
Firefox
|
159/183 24 timed out | Z.ai | AWS Bedrock |
| 5 |
Kimi K2.5 (high)
Open
|
2.2%
4/183
V8
Firefox
|
180/183 3 timed out | Moonshot AI | AWS Bedrock |
| 6 |
MiniMax M2.5 (high)
Open
|
0.0%
0/183
V8
Firefox
|
176/183 7 timed out | MiniMax | AWS Bedrock |
Newer results for Claude Opus and Mythos models are not available due to safeguard restrictions.
Overview
SEC-bench Pro evaluates agents on vulnerability discovery and PoC generation tasks on challenging targets. Each target instance ships with a Docker image, metadata, a rendered prompt, and a harness. The harness renders the prompt from
meta.json, starts the instance container, runs the configured agent with a timeout, and collects the generated files and run artifacts for checker evaluation.
SEC-bench
Can LLM Agents Solve Critical Security Challenges?
Leaderboard
Performance of LLM agents on security engineering tasks.
PoC Generation evaluates agents on generating working security proof-of-concepts for real vulnerabilities.
News
- [Dec 2025]: OSS-Fuzz instances are now integrated into SEC-bench!
- [Sep 2025]: SEC-bench has been accepted to NeurIPS 2025!
- [June 2025]: We release the SEC-bench benchmark with PoC Generation and Vulnerability Patching tasks.
SEC-bench Overview
SEC-bench is the first fully automated benchmarking framework for evaluating LLM agents on authentic security engineering tasks.
PoC Generation Task
PoC Generation evaluates the ability of LLM agents to craft proof-of-concept (PoC) artifacts that trigger specified vulnerabilities. A submission is considered successful only if it creates a working PoC artifact that triggers a valid sanitizer error.
Evaluation Modes:
SEC-bench supports three evaluation modes with varying levels of information provided to agents:
Evaluation Modes:
SEC-bench supports three evaluation modes with varying levels of information provided to agents:
| Mode | Information Provided | Description |
|---|---|---|
poc-repo |
Repository only | Agent receives only the vulnerable code. Must discover and validate the bug from scratch. |
poc-desc |
Repository + Description | Agent receives repository plus high-level vulnerability description. |
poc-san |
Repository + Description + Sanitizer Report | Agent receives all available information including crash stack trace. |
Note: Current leaderboard results are evaluated using the poc-san mode. Read more about evaluation modes in our blog post.