EviRepair: Evidence-Driven Agents for Reliable Vulnerability Repair

Yilian Shi, Qingyu Liu, Tianyi Zhou, Wende Tan, Chenyang Si, Yueming Lyu, Xia Hu, Dongrui LiuarXiv 2026 · arXiv

TL;DR

EviRepair is a new multi-agent framework that significantly improves the reliability of automated software vulnerability repair. By introducing a rigorous "evidence review" process and separating vulnerability diagnosis from patch synthesis, it achieves a 16.37% improvement on PatchEval and a massive 40.33% boost on SecBench compared to existing state-of-the-art agentic baselines.

What Problem Does This Paper Address?

Automated Vulnerability Repair (AVR) is the "holy grail" of software security—having AI automatically find and fix bugs before hackers can exploit them. While Large Language Models (LLMs) have shown promise, they often struggle with repository-level context.

Current approaches fall into two traps:

  1. Single-agent systems suffer from "context overload," trying to handle diagnosis, generation, and testing all at once, which leads to messy, unreliable reasoning.
  2. Existing multi-agent systems often just aggregate outputs from different agents without checking if those outputs actually make sense together. If one agent misunderstands the bug, that error propagates through the entire pipeline, resulting in a "fix" that doesn't actually address the root cause.
Comparison of single-agent, conventional multi-agent, and EviRepair

Methods

The authors propose EviRepair, which treats vulnerability repair as an evidence-driven process. The framework is split into two distinct stages connected by a structured Vulnerability Model.

1. Vulnerability Diagnosis

Instead of one agent doing everything, an Analysis Coordinator dispatches three specialized agents in parallel:

  • Reachability Analyzer: Traces how attacker input moves from the entry point to the crash site.
  • Vulnerability Reproducer: Builds a script to trigger the bug and observes the failure.
  • Boundary Analyzer: Identifies what inputs should be blocked and what legitimate behaviors must be preserved.

The "secret sauce" is the Evidence Consistency Reviewer. It acts as an adversarial judge, challenging the findings. It ensures that the evidence from all three agents is consistent and actually explains the reported vulnerability. If things don't add up, it sends the agents back to do more work.

2. Patch Synthesis

Once the evidence is verified, it's assembled into a Vulnerability Model. This model serves as a strict specification for the second stage. A Repair Coordinator translates this model into a Validation Matrix—a checklist of tests the patch must pass. A Patch Synthesizer generates the fix, which is then independently audited by a Patch Reviewer to ensure it doesn't just "hide" the bug or break existing features.

Figure 2: Overview of EVIREPAIR, an evidence-driven multi-agent framework for AVR. Vulnerability Diag- nosis gathers complementary vulnerability evidence via specialized agents and validates their consistency to construct a Vulnerability Model. Patch Synthesis treats this model as the specification to generate, validate, and refine candidate patches.

Datasets

The researchers evaluated EviRepair on two major benchmarks:

  • PatchEval: 230 real-world CVE repair tasks across Go, Python, and JavaScript. It uses hidden tests to ensure patches are truly correct.
  • SecBench: 300 security tasks (200 CVEs and 100 OSS-Fuzz tasks). This benchmark provides more auxiliary evidence (like sanitizer logs), testing how well agents use provided data.

Key Insights and Ideas

  • "Agreement is not evidence": A core philosophy of EviRepair is that just because two agents agree doesn't mean they are right. The system requires inspectable evidence (logs, traces, reproduction scripts) that can be independently verified.
  • Structured Evidence as a Reasoning Booster: One of the most striking findings is that providing a structured evidence framework allows smaller, less capable LLMs to outperform much larger models that lack such a framework.
  • Decoupling Diagnosis from Synthesis: By forcing the system to fully "understand" the bug and document it in a Vulnerability Model before writing a single line of patch code, EviRepair prevents the "hallucination" of fixes for misunderstood problems.
Performance across different LLM backbones

Key Results

EviRepair was tested using three different LLM "backbones": DeepSeek-V4.1-Flash, GLM-5.2, and Qwen3.8-27B.

  • Superior Performance: It consistently outperformed baselines like OpenAI's Codex and Anomaly's OpenCode. On SecBench, EviRepair achieved a resolved rate of 93.11% (averaged across models), compared to 83.33% for the next best baseline.
  • Handling Complexity: The system showed its greatest strength in "hard" categories like authorization flaws and use-after-free bugs, where the fix is often located far from where the crash actually happens.
  • Ablation Success: Removing the multi-agent diagnosis stage dropped performance by nearly 4%, and removing the multi-agent synthesis stage dropped it by over 5%. Removing both caused a massive 12% performance hit, proving that the two-stage interaction is critical.
Resolved rate across vulnerability categories

Limitations and Future Work

While highly effective, EviRepair has a few hurdles to clear:

  • Multi-file Fixes: The current Vulnerability Model assumes a single root cause. The system struggled with complex patches that required coordinated changes across four or more files.
  • Cost and Latency: Because it uses multiple agents and iterative review cycles, EviRepair is more computationally expensive and slower than simpler single-agent approaches.
  • Sandbox Isolation: The evaluation was done in a network-isolated environment. Some vulnerabilities (like Server-Side Request Forgery) are hard to reproduce without network access, leading to weaker evidence collection in those specific cases.

Future work aims to extend the model to handle multiple interacting root causes and to optimize the trade-off between repair quality and resource cost.

Summary updated

Abstract

Recent advances in large language models (LLMs) have enabled agent-based approaches for repository-level automated vulnerability repair (AVR), showing promising repair capabilities. However, an important issue has received limited attention: how can we ensure that the evidence generated during the repair process is reliable enough to support accurate vulnerability understanding and patch generation? In this paper, we propose EviRepair, an evidence-driven multi-agent framework for repository-level AVR. EviRepair introduces evidence review into the repair process by enabling agents to collaboratively collect vulnerability evidence, verify evidence consistency, and construct a reviewed vulnerability model for patch synthesis and validation. Extensive experiments on PatchEval and SecBench benchmarks with multiple agent frameworks and LLMs demonstrate that EviRepair consistently outperforms existing agent-based baselines, achieving average improvements of 16.37% and 40.33% in resolved rates on PatchEval and SecBench, respectively. Codes are available at https://github.com/lly-s/EviRepair
BibTeX
@misc{shi2026evirepair,
  author = {Yilian Shi and Qingyu Liu and Tianyi Zhou and Wende Tan and Chenyang Si and Yueming Lyu and Xia Hu and Dongrui Liu},
  title = {{EviRepair: Evidence-Driven Agents for Reliable Vulnerability Repair}},
  year = {2026},
  eprint = {2610.13491},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.13491},
}

Loading related content...

Figure