TL;DR
BugGen 2.0 optimizes hardware bug synthesis by offloading structural analysis and target selection from Large Language Models (LLMs) to deterministic algorithms. This hybrid approach slashes inference costs by up to 15x and significantly reduces simulation overhead while generating a more diverse and valid set of RTL bugs for verification datasets.
What Problem Does This Paper Address?
In the world of Integrated Circuit (IC) development, functional verification and debugging consume more than half of the total design effort. To improve these processes—especially when training AI-assisted debugging tools—researchers need massive datasets of "ground-truth" bugs: code mutations with known locations and observable failures.
Manually creating these datasets is impossible at scale. While early LLM-based approaches like BugGen 1.0 showed promise in generating context-aware bugs, they were inefficient. They relied on the LLM to decide both where to mutate and how to mutate. This often led to "hallucinated" targets that failed compilation, wasting expensive inference tokens and slow hardware simulation cycles. BugGen 2.0 aims to fix this "division of labor" to make bug synthesis truly scalable.
Methods
The core philosophy of BugGen 2.0 is to let deterministic algorithms handle the rigid structural rules of hardware description languages (RTL), leaving the LLM to do what it does best: localized semantic rewriting.

Key components of the new workflow include:
- Type-Aware Filtering: Before the LLM is even called, the system uses regex and Abstract Syntax Tree (AST) analysis to identify compatible mutation targets. For example, it won't try to swap signals with mismatched bit-widths.
- Stratified Sampling: To ensure bugs are spread across the entire design rather than clustered in one area, the tool divides files into "strata" and samples from the least-used regions.
- Compile Learning: The system maintains a "penalty score" for different mutation types. If a specific type of bug consistently fails to compile in a certain module, the system learns to stop suggesting it.
- Variant Augmentation: Once a valid bug location is found, the system generates multiple alternative mutations at that same spot. To save time, it only simulates a subset of these variants and uses a "yield cache" to estimate the success rate of the rest.
Datasets
The authors evaluated BugGen 2.0 across five diverse hardware designs:
- OpenTitan IP Blocks: AES (Advanced Encryption Standard), OTP Controller, ROM Controller, and USB.
- FabScalar: A complex, synthesizable out-of-order superscalar processor core.
The evaluation compared BugGen 2.0 against its predecessor (BugGen 1.0) and Firefly, another LLM-based mutation tool, using three different LLM backends: Google Gemma 4 31B, OpenAI GPT-4 (oss-120b), and DeepSeek v4 Flash.
Key Insights and Ideas
- Don't Ask LLMs to be Compilers: LLMs struggle with the strict structural constraints of RTL. By using deterministic "pre-checks," BugGen 2.0 ensures that the LLM only works on valid candidates, drastically increasing the "yield" of successful bugs.
- Simulation is the Real Bottleneck: In hardware EDA, running a testbench (simulation) is often more expensive than an LLM call. The "Variant Augmentation" strategy is a clever way to expand a dataset by 3-4x while skipping up to 63% of the required simulations through statistical estimation.
- Model Agnosticism: Because the pipeline handles the structural heavy lifting, BugGen 2.0's performance is remarkably consistent across different LLMs. It doesn't require extensive prompt engineering or model-specific tuning to achieve high-quality results.
Key Results
BugGen 2.0 outperformed previous methods across nearly every metric:
- Token Efficiency: It reduced token usage by 6.8x–15.4x compared to BugGen 1.0 and 1.5x–7.9x compared to Firefly.
- Diversity: It successfully generated bugs for 14–16 out of 16 configured mutation types (e.g., bitwise corruption, FSM transition errors, signal swaps).
- Convergence Speed: As shown in the "Accepted Scenarios Over Time" plots, BugGen 2.0 reaches target bug quotas much faster than competing methods.
- Estimation Accuracy: The system's ability to estimate the success of unsimulated variants was highly accurate, showing only a 1.09% error rate on the FabScalar dataset.
Limitations and Future Work
While BugGen 2.0 excels at generating compilable bugs that cause test failures, the authors note that it doesn't yet characterize the architectural impact of those bugs. A bug might cause a failure, but does it represent a deep logic flaw or just a trivial syntax error?
Future work aims to incorporate "propagation-aware analysis" to distinguish between bugs that merely trigger failures and those with deeper functional effects. This would move the field toward even more "informative" bug datasets that better reflect the challenges of real-world hardware debugging.
Summary updated