MOS: Towards Effective Smart Contract Vulnerability Detection through Mixture-of-Experts Tuning of Large Language Models

Hang Yuan, Lei Yu, Zhirong Huang, Jingyuan Zhang, Junyi Lu, Shiqi Cheng, Li Yang, Fengjun Zhang, Jiajia Ma, Chun ZuoarXiv 2025 · DOI · OpenReview

TL;DR

This paper introduces MOS, a novel framework that significantly improves smart contract vulnerability detection by using a Mixture-of-Experts (MoE) approach with Large Language Models (LLMs). MOS not only achieves state-of-the-art detection accuracy but also provides high-quality, human-understandable explanations for identified vulnerabilities, addressing key limitations of existing methods.

What Problem Does This Paper Address?

Smart contracts, the backbone of many blockchain applications, are susceptible to vulnerabilities that can lead to catastrophic financial losses, as exemplified by the infamous DAO hack. Existing vulnerability detection methods have significant drawbacks:

  • Program Analysis: These methods rely on predefined patterns, making them inflexible and unable to adapt to new or evolving vulnerability types. They also struggle with complex vulnerabilities and often have limited coverage.
  • Deep Learning: While capable of detecting vulnerabilities, these methods often fail to provide clear explanations for their decisions, hindering debugging and trust.
  • Large Language Models (LLMs): LLMs show promise due to their natural language understanding, but they suffer from high false positive rates, inconsistent performance across different vulnerability types, and a lack of precise localization of issues.

The core challenge is to develop a system that is both highly accurate in detecting a wide range of vulnerabilities and provides clear, actionable explanations, overcoming the limitations of current approaches.

Methods

MOS tackles these challenges by leveraging a Mixture-of-Experts (MoE) tuning approach applied to Large Language Models (LLMs). The key innovations include:

  • Continual Pre-training: The base LLM undergoes specialized pre-training on a large corpus of smart contract code. This "domain-enhances" the model, providing a strong foundation for understanding contract-specific patterns and concepts.
  • Vulnerability-Aware Routing: A novel routing mechanism analyzes smart contract code features to dynamically activate the most relevant "expert" networks within the MoE architecture. This allows the system to adapt to specific vulnerability types without relying on rigid, predefined rules.
  • Specialized Mixture of Experts Network: The feed-forward layers of the LLM are transformed into multiple parallel "expert" networks. Each expert is trained to specialize in detecting and explaining specific types of vulnerabilities (e.g., reentrancy, integer overflow).
  • Dual-Objective Loss Function: The training process uses a loss function that optimizes both vulnerability detection and explanation quality, while also encouraging a balanced distribution of vulnerability types across experts.
  • High-Quality Data Construction: A multi-stage pipeline involving LLM generation and expert verification is used to create a dataset of vulnerability explanations, ensuring their accuracy, completeness, and conciseness.

Datasets

  • Continual Pre-training Dataset: A large dataset comprising 186,397 unique smart contract instances from Ethereum, augmented with additional data from general code, mathematics, English, and Chinese text.
  • MOE-Tuning Dataset: A curated dataset of 6,524 high-quality vulnerability explanations, specifically designed for training the MoE model. This dataset was built by combining LLM generation with expert review and covers four key vulnerability types: reentrancy, timestamp dependency, integer overflow/underflow, and delegatecall.

Key Insights and Ideas

  • Expert Specialization is Key: The MoE architecture, guided by vulnerability-aware routing, effectively creates specialized "experts" within the LLM that can deeply understand and detect specific vulnerability patterns. This is a significant departure from monolithic models.
  • Routing Matters: The vulnerability-aware routing mechanism is crucial for directing the right code features to the right experts, preventing the model from becoming overwhelmed by general code semantics and enabling focused analysis.
  • Domain-Specific Pre-training is Essential: Simply fine-tuning a general LLM is insufficient. Continual pre-training on smart contract code provides the necessary foundational knowledge for effective vulnerability detection.
  • Explanations are as Important as Detection: The MOS framework's emphasis on generating high-quality, human-evaluable explanations is a critical step towards building trust and enabling practical adoption of automated security tools.
  • Balancing is Crucial for MoE: The dual-objective loss function, including a load-balancing component, is vital for ensuring that all experts are adequately trained and utilized, preventing performance degradation.

Key Results

  • State-of-the-Art Detection: MOS significantly outperforms 16 state-of-the-art baseline methods across four key vulnerability types. It achieved average improvements of 6.32% in F1 score and 4.80% in accuracy compared to existing methods.
  • Balanced Performance: MOS demonstrates a more balanced detection performance across the four vulnerability types compared to baselines, with a significantly lower standard deviation.
  • High-Quality Explanations: MOS-generated explanations received positive ratings (scores of 3-4 on a 4-point scale) of 82.96% for correctness, 85.21% for completeness, and 94.58% for conciseness, as evaluated by both human experts and an advanced LLM.
  • Expert Specialization Observed: Analysis of the model's internal workings revealed a hierarchical pattern of expert specialization across different layers, with deeper layers showing more pronounced differentiation for specific vulnerabilities.

Limitations and Future Work

  • Computational Requirements: Training MOS requires substantial computational resources, including high-end GPUs with significant VRAM. While inference is more efficient due to sparse activation, the total model parameters are still large.
  • Prompt Engineering: The authors acknowledge that prompt engineering is an area for improvement. Exploring alternative prompt formulations, modified instruction structures, and few-shot examples could further enhance performance.
  • Expanding Vulnerability Coverage: The authors plan to focus on expanding the types of vulnerabilities covered in future research.
  • Addressing False Positives/Negatives: While MOS significantly reduces errors, the paper notes that it still tends to generate false positives for secure patterns and false negatives in complex scenarios, indicating room for improvement in accuracy.

Summary updated

Abstract

Smart contract vulnerabilities pose significant security risks to blockchain systems, potentially leading to severe financial losses. Existing methods face several limitations: (1) Program analysis-based approaches rely on predefined patterns, lacking flexibility for new vulnerability types; (2) Deep learning-based methods lack explanations; (3) Large language model-based approaches suffer from high false positives. We propose MOS, a smart contract vulnerability detection framework based on mixture-of-experts tuning (MOE-Tuning) of large language models. First, we conduct continual pre-training on a large-scale smart contract dataset to provide domain-enhanced initialization. Second, we construct a high-quality MOE-Tuning dataset through a multi-stage pipeline combining LLM generation and expert verification for reliable explanations. Third, we design a vulnerability-aware routing mechanism that activates the most relevant expert networks by analyzing code features and their matching degree with experts. Finally, we extend the feed-forward layers into multiple parallel expert networks, each specializing in specific vulnerability patterns. We employ a dual-objective loss function: one for optimizing detection and explanation performance, and another for ensuring reasonable distribution of vulnerability types to experts through entropy calculation. Experiments show that MOS significantly outperforms existing methods with average improvements of 6.32% in F1 score and 4.80% in accuracy. The vulnerability explanations achieve positive ratings (scores of 3-4 on a 4-point scale) of 82.96%, 85.21% and 94.58% for correctness, completeness, and conciseness through human and LLM evaluation.
BibTeX
@misc{yuan2025mos,
  author = {Hang Yuan and Lei Yu and Zhirong Huang and Jingyuan Zhang and Junyi Lu and Shiqi Cheng and Li Yang and Fengjun Zhang and Jiajia Ma and Chun Zuo},
  title = {{MOS: Towards Effective Smart Contract Vulnerability Detection through Mixture-of-Experts Tuning of Large Language Models}},
  year = {2025},
  eprint = {2504.12234},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2504.12234},
}
Figure