VerilogMonkey: Exploring Parallel Scaling for Automated Verilog Code Generation with LLMs

Juxin Niu, Yuxin Du, Dan Niu, Xi Wang, Zhe Jiang, Nan GuanarXiv 2025 · DOI · arXiv

Abstract

Automated hardware description language (HDL) generation remains a significant challenge for Large Language Models (LLMs) due to the intricate requirements of digital logic and concurrency. This paper introduces VerilogMonkey, a framework that leverages parallel scaling—the practice of generating and testing hundreds of candidate solutions in parallel—to overcome these hurdles. Our results demonstrate that scaling inference to 512 samples allows general-purpose models like Claude 3.7 Sonnet to achieve a state-of-the-art 94.87% success rate on the VerilogEval benchmark, outperforming specialized reasoning models. We show that this "brute-force" approach is not only highly effective but also cost-efficient and time-competitive when implemented with a concurrent execution pipeline.

Introduction

The design of modern integrated circuits relies heavily on Verilog, a hardware description language that demands precise timing and logic synchronization. While LLMs have revolutionized software engineering, their application to Verilog has been limited by high error rates and "hallucinations" in complex logic. Existing research typically focuses on training-time scaling (larger models) or sequential scaling (multi-turn agentic loops). However, these methods often incur high latency or computational costs.

We propose a third dimension: parallel scaling. By treating LLM generation as a probabilistic sampling problem, we can explore a vast solution space simultaneously. This paper investigates the limits of this approach using the VerilogMonkey framework.

Methodology: The VerilogMonkey Framework

VerilogMonkey is designed to manage the high-throughput requirements of parallel scaling. The framework consists of a modular, asynchronous pipeline that decouples LLM generation from hardware simulation.

The VerilogMonkey framework architecture, showing the controller managing task queues for parallel LLM generation and simulation.

The system operates by dispatching I/O-bound LLM requests in large batches. Once candidates are generated, they are fed into a multi-process simulation engine that utilizes tools such as Icarus Verilog or VCS. To analyze model behavior, we employ Mean Cosine Distance (MCD) to measure output dispersion and KMeans clustering to categorize failure modes, distinguishing between simple hallucinations and deep logic uncertainty.

Experimental Setup

We evaluated parallel scaling across three major benchmarks, totaling approximately 200 problems:

  • VerilogEval: A standard benchmark for basic Verilog tasks.
  • RTLLM (v2): Focused on more complex Register-Transfer Level (RTL) designs.
  • GenBen: A generative benchmark for hardware design.

Testing involved a suite of state-of-the-art models, including GPT-4o, Gemini 1.5/2.0 Flash, and Claude 3.5/3.7 Sonnet.

Key Results and Discussion

Our experiments reveal that parallel scaling follows a sub-logarithmic trend ($C \propto \log \log N$). Increasing the sample count to 512 significantly boosts the success rate. For instance, Claude 3.7 Sonnet reached a 94.87% success rate on VerilogEval (Hit@512), setting a new state-of-the-art benchmark.

Efficiency and Concurrency

A common critique of parallel sampling is the perceived computational overhead. However, VerilogMonkey demonstrates that wall-clock time remains low through effective concurrency.

Performance analysis showing that concurrent execution significantly reduces time per job as thread count increases, making large-scale sampling practical.

As shown in the performance metrics, increasing the number of threads allows the system to process hundreds of simulation jobs in nearly the same time as a single sequential job. Furthermore, the cost of generating 500 samples (ranging from $0.04 to $10) is negligible compared to the hourly rate of a professional hardware engineer.

Escaping Hallucinations vs. Navigating Uncertainty

Parallel scaling is particularly effective at "escaping" model hallucinations. In simpler problems where a model might consistently produce a specific wrong pattern, a high sample count provides the necessary "rolls of the dice" to find the correct implementation. In more complex mathematical problems, scaling acts as an explorer, navigating the model's inherent uncertainty to identify valid logic paths.

Conclusion

VerilogMonkey proves that parallel scaling is a powerful, underutilized tool for hardware design. By shifting the complexity from model training or sequential reasoning to parallel inference and verification, we can achieve superior results with existing LLMs. Future work will explore hybrid scaling models that combine the breadth of parallel sampling with the depth of agentic reflection.

Summary updated

Abstract

We present VerilogMonkey, an empirical study of parallel scaling for the under-explored task of automated Verilog generation. Parallel scaling improves LLM performance by sampling many outputs in parallel. Across multiple benchmarks and mainstream LLMs, we find that scaling to hundreds of samples is cost-effective in both time and money and, even without any additional enhancements such as post-training or agentic methods, surpasses prior results on LLM-based Verilog generation. We further dissect why parallel scaling delivers these gains and show how output randomness in LLMs affects its effectiveness.
BibTeX
@misc{niu2025verilogmonkey,
  author = {Juxin Niu and Yuxin Du and Dan Niu and Xi Wang and Zhe Jiang and Nan Guan},
  title = {{VerilogMonkey: Exploring Parallel Scaling for Automated Verilog Code Generation with LLMs}},
  year = {2025},
  eprint = {2509.16246},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.16246},
}
Figure