LLM-BSCVM: An LLM-Based Blockchain Smart Contract Vulnerability Management Framework

Yanli Jin, Chunpei Li, Peng Fan, Peng Liu, Xianxian Li, Chen Liu, Wangjie QiuarXiv 2025 · DOI · arXiv

Abstract

Smart contracts are the backbone of the Web 3.0 ecosystem, but their "code is law" nature means a single bug can lead to catastrophic financial loss. While detection tools have improved, simply identifying vulnerabilities is insufficient for developers who require comprehensive management. This paper introduces LLM-BSCVM, a framework that doesn't just detect vulnerabilities—it analyzes, risks, repairs, and verifies them using a collaborative team of AI agents. By coordinating six specialized agents and leveraging Retrieval-Augmented Generation (RAG), LLM-BSCVM achieves over 91% detection accuracy while providing automated code repairs and detailed audit reports.

Introduction

The security of decentralized finance (DeFi) is currently a game of cat and mouse. Traditional tools, such as symbolic execution or fuzzing, often suffer from high false-positive rates or struggle with complex logic. While Large Language Models (LLMs) have shown promise in code analysis, most existing research treats them as simple "bug hunters." Vulnerability management is a multi-dimensional problem: finding a bug is only the first step. Developers also need to understand the root cause, assess the risk level, implement a repair, and verify the fix. LLM-BSCVM addresses this gap by covering the entire vulnerability lifecycle.

Methodology: The Decompose-Retrieve-Generate Workflow

The core of LLM-BSCVM is a three-stage process designed to maximize accuracy and minimize hallucinations.

The LLM-BSCVM framework architecture following the Decompose-Retrieve-Generate workflow.

1. Task Decomposition

The complex job of security auditing is broken down into six sub-tasks, each handled by a specialized "Intelligent Agent":

  • Detector: Identifies potential vulnerabilities using a mix of static analysis and LLM inference.
  • Advisor: Analyzes the root cause and suggests repair strategies.
  • Assessor: Assigns a risk level (Critical to Low) based on potential impact.
  • Fixer: Generates the actual repaired Solidity code.
  • Verifier: Acts as an independent auditor to check if the fix is valid.
  • Reporter: Compiles findings into a comprehensive, human-readable audit report.

2. Knowledge Retrieval (RAG)

To prevent hallucinations, agents utilize Retrieval-Augmented Generation (RAG) to query external databases, including a Smart Contract Corpus and a Vulnerability Knowledge Base containing 263 expert audit reports.

Detailed multi-agent pipeline for vulnerability detection, assessment, and repair.

3. Result Generation

The agents combine retrieved context with fine-tuned reasoning to produce explainable results rather than binary outputs. This specialization mimics a real-world security firm, creating a check-and-balance mechanism that improves reliability.

Key Results and Insights

The framework was evaluated using datasets from TrustLLM and Dappscan. Key findings include:

  • High Accuracy: The framework achieved a vulnerability detection accuracy and F1 score of over 91%, outperforming base models like CodeLlama by nearly 40 percentage points.
  • Superior Precision: It reached a precision of 94.95%, significantly reducing the False Positive Rate (FPR) to 5.1%.
  • Automated Repair: Approximately 21% of contracts were successfully repaired to a level that passed independent verification.
  • The Importance of RAG: Ablation studies confirmed that RAG is the most critical component; without external knowledge retrieval, performance dropped significantly.

Conclusion

LLM-BSCVM represents a significant step forward in automated smart contract security. By moving beyond simple detection to a full management lifecycle, it provides developers with actionable insights and verified repairs. Future work will focus on integrating formal verification for mathematical proofs of security and optimizing human-in-the-loop interactions.

Summary updated

Abstract

Smart contracts are a key component of the Web 3.0 ecosystem, widely applied in blockchain services and decentralized applications. However, the automated execution feature of smart contracts makes them vulnerable to potential attacks due to inherent flaws, which can lead to severe security risks and financial losses, even threatening the integrity of the entire decentralized finance system. Currently, research on smart contract vulnerabilities has evolved from traditional program analysis methods to deep learning techniques, with the gradual introduction of Large Language Models. However, existing studies mainly focus on vulnerability detection, lacking systematic cause analysis and Vulnerability Repair. To address this gap, we propose LLM-BSCVM, a Large Language Model-based smart contract vulnerability management framework, designed to provide end-to-end vulnerability detection, analysis, repair, and evaluation capabilities for Web 3.0 ecosystem. LLM-BSCVM combines retrieval-augmented generation technology and multi-agent collaboration, introducing a three-stage method of Decompose-Retrieve-Generate. This approach enables smart contract vulnerability management through the collaborative efforts of six intelligent agents, specifically: vulnerability detection, cause analysis, repair suggestion generation, risk assessment, vulnerability repair, and patch evaluation. Experimental results demonstrate that LLM-BSCVM achieves a vulnerability detection accuracy and F1 score exceeding 91\% on benchmark datasets, comparable to the performance of state-of-the-art (SOTA) methods, while reducing the false positive rate from 7.2\% in SOTA methods to 5.1\%, thus enhancing the reliability of vulnerability management. Furthermore, LLM-BSCVM supports continuous security monitoring and governance of smart contracts through a knowledge base hot-swapping dynamic update mechanism.
BibTeX
@misc{jin2025llm,
  author = {Yanli Jin and Chunpei Li and Peng Fan and Peng Liu and Xianxian Li and Chen Liu and Wangjie Qiu},
  title = {{LLM-BSCVM: An LLM-Based Blockchain Smart Contract Vulnerability Management Framework}},
  year = {2025},
  eprint = {2505.17416},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2505.17416},
}
Figure