TL;DR
EviRepair is a new multi-agent framework that significantly improves the reliability of automated software vulnerability repair. By introducing a rigorous "evidence review" process and separating vulnerability diagnosis from patch synthesis, it achieves a 16.37% improvement on PatchEval and a massive 40.33% boost on SecBench compared to existing state-of-the-art agentic baselines.
What Problem Does This Paper Address?
Automated Vulnerability Repair (AVR) is the "holy grail" of software security—having AI automatically find and fix bugs before hackers can exploit them. While Large Language Models (LLMs) have shown promise, they often struggle with repository-level context.
Current approaches fall into two traps:
- Single-agent systems suffer from "context overload," trying to handle diagnosis, generation, and testing all at once, which leads to messy, unreliable reasoning.
- Existing multi-agent systems often just aggregate outputs from different agents without checking if those outputs actually make sense together. If one agent misunderstands the bug, that error propagates through the entire pipeline, resulting in a "fix" that doesn't actually address the root cause.
Methods
The authors propose EviRepair, which treats vulnerability repair as an evidence-driven process. The framework is split into two distinct stages connected by a structured Vulnerability Model.
1. Vulnerability Diagnosis
Instead of one agent doing everything, an Analysis Coordinator dispatches three specialized agents in parallel:
- Reachability Analyzer: Traces how attacker input moves from the entry point to the crash site.
- Vulnerability Reproducer: Builds a script to trigger the bug and observes the failure.
- Boundary Analyzer: Identifies what inputs should be blocked and what legitimate behaviors must be preserved.
The "secret sauce" is the Evidence Consistency Reviewer. It acts as an adversarial judge, challenging the findings. It ensures that the evidence from all three agents is consistent and actually explains the reported vulnerability. If things don't add up, it sends the agents back to do more work.
2. Patch Synthesis
Once the evidence is verified, it's assembled into a Vulnerability Model. This model serves as a strict specification for the second stage. A Repair Coordinator translates this model into a Validation Matrix—a checklist of tests the patch must pass. A Patch Synthesizer generates the fix, which is then independently audited by a Patch Reviewer to ensure it doesn't just "hide" the bug or break existing features.
Datasets
The researchers evaluated EviRepair on two major benchmarks:
- PatchEval: 230 real-world CVE repair tasks across Go, Python, and JavaScript. It uses hidden tests to ensure patches are truly correct.
- SecBench: 300 security tasks (200 CVEs and 100 OSS-Fuzz tasks). This benchmark provides more auxiliary evidence (like sanitizer logs), testing how well agents use provided data.
Key Insights and Ideas
- "Agreement is not evidence": A core philosophy of EviRepair is that just because two agents agree doesn't mean they are right. The system requires inspectable evidence (logs, traces, reproduction scripts) that can be independently verified.
- Structured Evidence as a Reasoning Booster: One of the most striking findings is that providing a structured evidence framework allows smaller, less capable LLMs to outperform much larger models that lack such a framework.
- Decoupling Diagnosis from Synthesis: By forcing the system to fully "understand" the bug and document it in a Vulnerability Model before writing a single line of patch code, EviRepair prevents the "hallucination" of fixes for misunderstood problems.
Key Results
EviRepair was tested using three different LLM "backbones": DeepSeek-V4.1-Flash, GLM-5.2, and Qwen3.8-27B.
- Superior Performance: It consistently outperformed baselines like OpenAI's Codex and Anomaly's OpenCode. On SecBench, EviRepair achieved a resolved rate of 93.11% (averaged across models), compared to 83.33% for the next best baseline.
- Handling Complexity: The system showed its greatest strength in "hard" categories like authorization flaws and use-after-free bugs, where the fix is often located far from where the crash actually happens.
- Ablation Success: Removing the multi-agent diagnosis stage dropped performance by nearly 4%, and removing the multi-agent synthesis stage dropped it by over 5%. Removing both caused a massive 12% performance hit, proving that the two-stage interaction is critical.
Limitations and Future Work
While highly effective, EviRepair has a few hurdles to clear:
- Multi-file Fixes: The current Vulnerability Model assumes a single root cause. The system struggled with complex patches that required coordinated changes across four or more files.
- Cost and Latency: Because it uses multiple agents and iterative review cycles, EviRepair is more computationally expensive and slower than simpler single-agent approaches.
- Sandbox Isolation: The evaluation was done in a network-isolated environment. Some vulnerabilities (like Server-Side Request Forgery) are hard to reproduce without network access, leading to weaker evidence collection in those specific cases.
Future work aims to extend the model to handle multiple interacting root causes and to optimize the trade-off between repair quality and resource cost.
Summary updated