SolEval: Benchmarking Large Language Models for Repository-level Solidity Smart Contract Generation

Zhiyuan Peng, Xin Yin, Rui Qian, Peiqin Lin, YongKang Liu, Hao Zhang, Chenhao Ying, Yuan LuoEMNLP 2025

TL;DR

This paper introduces SolEval, the first repository-level benchmark for evaluating Large Language Models (LLMs) on Solidity smart contract generation. They found that current LLMs struggle significantly with this task, highlighting a substantial gap in their capabilities for generating secure and efficient smart contracts.

What Problem Does This Paper Address?

The rapid growth of blockchain technology and Decentralized Finance (DeFi) has led to an explosion in smart contract development. However, generating secure and cost-effective smart contracts is a complex challenge. While LLMs have shown promise in code generation for mainstream languages like Python, their performance on Solidity, the primary language for Ethereum smart contracts, remains largely unexplored due to a lack of comprehensive benchmarks. Existing Solidity benchmarks are often limited in scale, focus only on standalone functions, and don't reflect the real-world complexity of repository-level development, which involves inter-file dependencies and context.

Methods

The authors introduce SolEval, a new benchmark designed to evaluate LLMs on repository-level Solidity smart contract generation. This involves:

  • Project Selection: Crawling and filtering popular GitHub Solidity repositories to ensure real-world relevance.
  • Function Parsing: Using a custom Solidity parser to extract functions, their signatures, requirements, and dependencies.
  • Test Construction: Generating comprehensive unit tests for each function to ensure accurate evaluation of correctness.
  • Human Annotation: Manually annotating function descriptions to provide high-quality, unambiguous requirements for LLMs, mitigating potential pre-training data leakage.
  • Contextualization: Incorporating repository context (other functions, variables, interfaces) into the prompts, crucial for generating non-standalone functions.
  • New Metrics: Introducing Gas@k and Vul@k metrics alongside traditional Pass@k and Compile@k to assess gas efficiency and vulnerability, respectively.

Datasets

The primary dataset introduced is SolEval, comprising 1,507 samples from 28 real-world Solidity repositories across 6 domains. This benchmark is designed to be repository-level, meaning it considers functions within their broader project context, unlike previous benchmarks that focused on isolated functions.

Key Insights and Ideas

  • Solidity is Hard for LLMs: Even the best-performing LLMs struggle significantly with Solidity code generation, achieving only 26.29% Pass@10. This indicates a substantial gap in current LLM capabilities for this domain.
  • Repository Context Matters: Generating code within the context of a larger repository, including dependencies and inter-file calls, is significantly more challenging than standalone function generation.
  • Balancing Act: There's a fundamental trade-off in Solidity code generation between functional correctness (Pass@k) and gas efficiency (Gas@k). Models that excel at one often falter at the other.
  • Fine-tuning Pays Off: Supervised fine-tuning (SFT) on SolEval dramatically improves LLM performance, with Qwen-7B seeing a jump from 16.67% Pass@5 to 58.33%.
  • RAG and Context Help: Retrieval-Augmented Generation (RAG) and providing relevant contextual information improve LLM performance, highlighting the importance of these techniques for complex code generation.

Key Results

  • LLM Performance: 16 LLMs were evaluated. The best performer, DeepSeek-V3, achieved a Pass@10 of 26.29%. Many models showed very low Pass@1 scores (e.g., DeepSeek-R1-Distill-Qwen-7B at 2.08%).
  • Gas Efficiency: DeepSeek-V3 also achieved the best Gas@1 (7.13%), indicating it generates more cost-efficient contracts. However, it also had higher gas fees overall compared to some other models.
  • Fine-tuning Impact: Supervised fine-tuning of Qwen-7B on SolEval led to a significant improvement, with Pass@5 increasing from 16.67% to 58.83% and Vul@1 decreasing by over 19 percentage points.

Limitations and Future Work

  • Monolingual Focus: SolEval currently only supports Solidity. Future work aims to develop a multilingual version to assess LLMs' capabilities across languages like Vyper and Rust.
  • Limited Model Evaluation: Due to funding constraints, the study couldn't evaluate the latest proprietary models like GPT-5 or Claude Opus 4.1.
  • Metrics for Optimization: While Gas@k and Vul@k evaluate efficiency and security, they don't provide mechanisms for optimization or remediation. Future work will explore methods for gas optimization and vulnerability detection.
  • Expanding the Benchmark: The authors plan to enlarge SolEval by incorporating more high-quality code repositories.
  • Advanced Fine-tuning: They intend to leverage SFT and Direct Preference Optimization (DPO) to fine-tune LLMs for safer and cheaper code generation.

Summary updated

BibTeX
@inproceedings{peng2025soleval,
  author = {Zhiyuan Peng and Xin Yin and Rui Qian and Peiqin Lin and YongKang Liu and Hao Zhang and Chenhao Ying and Yuan Luo},
  title = {{SolEval: Benchmarking Large Language Models for Repository-level Solidity Smart Contract Generation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure