VERIRL: Boosting the LLM-based Verilog Code Generation via Reinforcement Learning

Fu Teng, Miao Pan, Xuhong Zhang, Zhezhi He, Yiyao Yang, Xinyi Chai, Mengnan Qi, Liqiang Lu, Jianwei Yin

TL;DR

This paper introduces VERIRL, a novel reinforcement learning (RL) framework that significantly boosts Verilog hardware description language (HDL) code generation by LLMs. It tackles key challenges like sparse rewards and training instability with a new dataset, a "Trace-back based Rescore" mechanism, and a "Sample-balanced Weighting" strategy, achieving state-of-the-art results.

What Problem Does This Paper Address?

While Large Language Models (LLMs) have excelled in generating code for traditional software languages, they've struggled with Hardware Description Languages (HDLs) like Verilog. This is due to Verilog's unique complexities: concurrency, strict timing semantics, and the difficulty of verifying generated code through simulation. Existing methods often rely on massive, sometimes proprietary, datasets or struggle with the noisy and sparse feedback signals inherent in Verilog code generation, leading to models that memorize patterns rather than truly understanding hardware design principles. This gap hinders the application of powerful LLMs to the critical domain of hardware design.

Methods

VERIRL employs a multi-stage RL framework, inspired by recent LLM advancements, but with crucial domain-specific adaptations for Verilog:

  • Trace-back based Rescore (TbR): This mechanism addresses sparse and noisy rewards. Instead of discarding incorrect code, TbR iteratively refines it by analyzing reasoning paths and testbench feedback. It allows for the recovery and improvement of partially correct but syntactically flawed code, enriching the reward signal and improving preference data quality.
  • Sample-balanced Weighting (SbW): To combat catastrophic forgetting and overfitting during RL fine-tuning, SbW adaptively adjusts the influence of different training samples. It categorizes samples based on their generation probability and reward, assigning higher weights to rare but high-quality examples and lower weights to noisy or trivial ones. This stabilizes the learning process and promotes generalization.
  • Iterative RL Pipeline: The framework co-evolves policy and reward models, allowing for continuous improvement.

Datasets

  • Veribench-53K: A newly constructed, high-quality dataset for Verilog code generation. It's curated from over 700K Verilog problems, enriched with structured prompts, complexity labels, and diverse testbenches. This dataset is crucial for training reliable reward models and evaluating Verilog generation capabilities.
  • Existing Benchmarks: The paper also evaluates VERIRL on established Verilog benchmarks like VerilogEval-Human and RTLLM v1.1, demonstrating its effectiveness against prior work.

Key Insights and Ideas

  • Verilog's unique challenges require specialized RL: Standard RL techniques are insufficient for Verilog due to its simulation-based verification and complex semantics. Novel reward shaping and sample weighting are essential.
  • Iterative refinement of imperfect code is key: Instead of discarding code with minor errors, TbR's ability to recover and improve them significantly enhances the quality of training data and reward signals. This is a departure from simply filtering out bad outputs.
  • Balancing exploration and exploitation is critical: SbW's adaptive weighting strategy is vital for preventing LLMs from overfitting to noisy data or forgetting core Verilog principles during RL fine-tuning.
  • Smaller, high-quality datasets with RL can outperform larger, less curated ones: VERIRL demonstrates that a well-structured dataset combined with advanced RL techniques can achieve superior results compared to methods relying on massive, distilled datasets.

Key Results

VERIRL achieves state-of-the-art performance on Verilog code generation tasks. Specifically:

  • On the VerilogEval-Human benchmark, VERIRL (using CodeQwen2.5) achieves 69.3% pass@1 and 78.1% pass@5, outperforming previous leading models.
  • On the RTLLM v1.1 benchmark, VERIRL achieves 58.2% pass@1 and 66.0% pass@5.
  • The framework also shows strong generalization capabilities when applied to other models like Haven, boosting its performance.
  • Ablation studies confirm that both the TbR mechanism and the SbW strategy contribute significantly to the performance gains.

Limitations and Future Work

  • Limitations: The paper acknowledges that while VERIRL significantly improves Verilog code generation, challenges like ensuring absolute functional correctness and handling extremely complex designs remain areas for further research. The reliance on simulation for reward signals, though improved by TbR, can still be a bottleneck.
  • Future Work: The authors suggest exploring more advanced RL algorithms and reward modeling techniques tailored for HDL. Further investigation into generating more diverse and complex testbenches could also enhance evaluation robustness. The potential for VERIRL to be applied to other HDL languages is also an avenue for future exploration.

Summary updated

Abstract

Recent advancements in code generation have shown remarkable success across software domains, yet hardware description languages (HDLs) such as Verilog remain underexplored due to their concurrency semantics, syntactic rigidity, and simulation complexity. In this work, we address these challenges by introducing a reinforcement learning (RL) framework tailored for Verilog code generation. We first construct Veribench-53K, a high-quality dataset curated from over 700K Verilog problems, enriched with structured prompts, complexity labels, and diverse testbenches. To tackle the problem of sparse and noisy reward signals, we propose a Trace-back based Rescore mechanism that leverages reasoning paths and iterative refinement to enhance feedback reliability and support reward model training. Furthermore, to mitigate catastrophic forgetting and overfitting during RL fine-tuning, we introduce a sample-balanced weighting strategy that adaptively balances learning dynamics based on reward-probability distributions. These innovations are integrated into an iterative RL pipeline that co-evolves the policy and reward models. In contrast to recent work such as CraftRTL, which relies on large-scale closed-source model distillation, and DeepSeek-style approaches that struggle with sparse feedback, our method demonstrates superior performance using a smaller but high-quality dataset combined with RL optimization. Experiments on Verilog generation tasks demonstrate state-of-the-art performance, with substantial gains in test pass rate, functional correctness, and compilation robustness. Our findings highlight the potential of RL-driven approaches for structured code generation in hardware-centric domains. VERIRL is publicly available at https://github.com/omniAI-Lab/VeriRL.
BibTeX
@inproceedings{teng2025verirl,
  author = {Fu Teng and Miao Pan and Xuhong Zhang and Zhezhi He and Yiyao Yang and Xinyi Chai and Mengnan Qi and Liqiang Lu and Jianwei Yin},
  title = {{VERIRL: Boosting the LLM-based Verilog Code Generation via Reinforcement Learning}},
  booktitle = {ICCAD},
  year = {2025},
  doi = {10.1109/iccad66269.2025.11241003},
}
Figure