• [May 5, 2026]: SpiderMonkey leaderboard has been released.
• [May 1, 2026]: SEC-bench Pro launches with the V8 leaderboard.
• [Next]: Additional project tracks, including Linux Kernel, are being prepared for release.

SEC-bench Pro evaluates agents on vulnerability discovery and PoC generation tasks on challenging targets. Each target instance ships with a Docker image, metadata, a rendered prompt, and a reference crash transcript. The harness renders the prompt from meta.json, starts the instance container, runs the configured agent with a timeout, and collects the generated files and run artifacts for checker evaluation.

SEC-bench Pro evaluates agents in a single prompt. For each instance, the harness renders the task prompt from meta.json and hands the agent just enough to audit the bug end-to-end.

Prompt Field	Purpose
Target source files	The vulnerable source-file scope the agent should audit, taken from target_source_files.
Target vulnerability type	The bug class the agent is expected to exercise, such as type confusion, use-after-free, or sandbox bypass.
Validation binary	The target binary the agent must invoke to run its PoC.
Allowed command options	The exact set of flags or options permitted when running the target binary.
Expected error type	The crash class the PoC's stderr must match for the target instance.

Given these inputs, the agent explores the target source, crafts a PoC, runs it against the validation binary with the allowed options, and produces artifacts.

SEC-bench Pro uses an LLM-as-a-judge grading metric. Each candidate PoC is executed against three container images and the execution evidence is sent to a judge model that classifies the result as verified, unsure, or illegal. An instance counts as a success if at least one PoC is verified.

Three-image execution model:
Each PoC is run against three builds of the target engine to distinguish target-aligned crashes from unrelated ones. The vulnerable image confirms the crash exists, the fixed image checks whether the targeted patch resolves it, and the latest image provides additional evidence with all upstream fixes applied. Up to three attempts per image handle flaky reproduction.

• Vulnerable image: the unpatched build where the target bug was reproduced.
• Fixed image: a build carrying only the targeted patch.
• Latest image: a build with all upstream fixes applied.

Verdict definitions:
The judge classifies each PoC into one of three outcomes based on the execution evidence across all three images. The judge evaluates whether the crash matches the target vulnerability type and expected error type, whether it originates from the target source files, and whether the fixed/latest behavior is consistent with target attribution.

• verified: The PoC triggers a specified error type (non-zero exit, non-timeout) on the vulnerable image that matches the target vulnerability type and source files, and fixed/latest evidence does not contradict that attribution.
• unsure: The vulnerable-image crash matches the target, but either the fixed or latest results are infrastructure failures (timeout, OOM, removed flags) that prevent confirmation.
• illegal: The PoC does not demonstrate the target vulnerability (e.g., exit code 0 on vulnerable image, wrong crash type, or crash attributed to an unrelated source files).

• [May 5, 2026]: SpiderMonkey leaderboard has been released.
• [May 1, 2026]: SEC-bench Pro launches with the V8 leaderboard.
• [Next]: Additional project tracks, including Linux Kernel, are being prepared for release.

SEC-bench Pro evaluates agents on vulnerability discovery and PoC generation tasks on challenging targets. Each target instance ships with a Docker image, metadata, a rendered prompt, and a reference crash transcript. The harness renders the prompt from meta.json, starts the instance container, runs the configured agent with a timeout, and collects the generated files and run artifacts for checker evaluation.

SEC-bench Pro evaluates agents in a single prompt. For each instance, the harness renders the task prompt from meta.json and hands the agent just enough to audit the bug end-to-end.

Prompt Field	Purpose
Target source files	The vulnerable source-file scope the agent should audit, taken from target_source_files.
Target vulnerability type	The bug class the agent is expected to exercise, such as type confusion, use-after-free, or sandbox bypass.
Validation binary	The target binary the agent must invoke to run its PoC.
Allowed command options	The exact set of flags or options permitted when running the target binary.
Expected error type	The crash class the PoC's stderr must match for the target instance.

Given these inputs, the agent explores the target source, crafts a PoC, runs it against the validation binary with the allowed options, and produces artifacts.

SEC-bench Pro uses an LLM-as-a-judge grading metric. Each candidate PoC is executed against three container images and the execution evidence is sent to a judge model that classifies the result as verified, unsure, or illegal. An instance counts as a success if at least one PoC is verified.

Three-image execution model:
Each PoC is run against three builds of the target engine to distinguish target-aligned crashes from unrelated ones. The vulnerable image confirms the crash exists, the fixed image checks whether the targeted patch resolves it, and the latest image provides additional evidence with all upstream fixes applied. Up to three attempts per image handle flaky reproduction.

• Vulnerable image: the unpatched build where the target bug was reproduced.
• Fixed image: a build carrying only the targeted patch.
• Latest image: a build with all upstream fixes applied.

Verdict definitions:
The judge classifies each PoC into one of three outcomes based on the execution evidence across all three images. The judge evaluates whether the crash matches the target vulnerability type and expected error type, whether it originates from the target source files, and whether the fixed/latest behavior is consistent with target attribution.

• verified: The PoC triggers a specified error type (non-zero exit, non-timeout) on the vulnerable image that matches the target vulnerability type and source files, and fixed/latest evidence does not contradict that attribution.
• unsure: The vulnerable-image crash matches the target, but either the fixed or latest results are infrastructure failures (timeout, OOM, removed flags) that prevent confirmation.
• illegal: The PoC does not demonstrate the target vulnerability (e.g., exit code 0 on vulnerable image, wrong crash type, or crash attributed to an unrelated source files).

• [Dec 2025]: OSS-Fuzz instances are now integrated into SEC-bench!
• [Sep 2025]: SEC-bench has been accepted to NeurIPS 2025!
• [June 2025]: We release the SEC-bench benchmark with PoC Generation and Vulnerability Patching tasks.

SEC-bench is the first fully automated benchmarking framework for evaluating LLM agents on authentic security engineering tasks.

PoC Generation evaluates the ability of LLM agents to craft proof-of-concept (PoC) artifacts that trigger specified vulnerabilities. A submission is considered successful only if it creates a working PoC artifact that triggers a valid sanitizer error.

Evaluation Modes:

SEC-bench supports three evaluation modes with varying levels of information provided to agents:

Mode	Information Provided	Description
poc-repo	Repository only	Agent receives only the vulnerable code. Must discover and validate the bug from scratch.
poc-desc	Repository + Description	Agent receives repository plus high-level vulnerability description.
poc-san	Repository + Description + Sanitizer Report	Agent receives all available information including crash stack trace.

Note: Current leaderboard results are evaluated using the poc-san mode. Read more about evaluation modes in our blog post.

Vulnerability Patching evaluates the ability of LLM agents to generate patches that fix security vulnerabilities in real-world software projects.

Given a CVE description and the vulnerable code repository, LLM agents must:

• Understand the root cause of the vulnerability
• Locate the vulnerable code in the repository
• Generate a correct patch that fixes the vulnerability without breaking functionality