Towards LLM-based Root Cause Analysis of Hardware Design Failures

Siyu Qiu, Muzhi Wang, Raheel Afsharmazayejani, Mohammad Moradi Shahmiri, Benjamin Tan, Hammond Pearce

TL;DR

This paper explores whether Large Language Models (LLMs) can act as "expert assistants" by identifying the root causes of hardware design failures during synthesis and simulation. Using a newly curated dataset of 34 buggy scenarios, the researchers found that advanced reasoning models like OpenAI’s o3-mini can achieve 100% accuracy (Pass@5) in explaining these errors, particularly when bolstered by Retrieval-Augmented Generation (RAG).

What Problem Does This Paper Address?

In the digital hardware design lifecycle, debugging is a notorious bottleneck. When a design fails during synthesis (the process of turning code into hardware gates) or simulation (testing the logic), engineers are often met with cryptic error messages. Understanding the "why" behind a failure requires deep domain knowledge of Hardware Description Languages (HDLs) like Verilog and VHDL, as well as the specific quirks of Electronic Design Automation (EDA) tools.

While previous research has focused on using LLMs to generate hardware code, this paper shifts the focus to Root Cause Analysis (RCA). The goal is to support a "human-in-the-loop" workflow where the LLM explains the bug to the engineer, effectively acting as a tutor or a senior reviewer to speed up the repair process and upskill the designer.

Methods

The authors evaluated four state-of-the-art models: GPT-3.5-turbo, GPT-4o, o3-mini, and DeepSeek R1. They tested two primary prompting strategies:

  1. Baseline Prompting: Providing the LLM with the buggy code and the error message.
  2. Retrieval-Augmented Generation (RAG): Augmenting the prompt with relevant snippets from a custom database of hardware textbooks, tutorials, and tool manuals. The system retrieves the top 6 most relevant pieces of information to give the model extra context.

To evaluate the quality of the explanations, the authors manually graded the outputs based on four criteria: concept accuracy, lack of inaccuracies (factuality), relevance, and whether the answer was correct & complete.

Datasets

The researchers introduced a new, open-source benchmark: root-cause-bench.

  • Size: 34 manually labeled hardware design errors.
  • Categories: 19 synthesis-time errors (syntax, type mismatches, illegal assignments) and 15 simulation runtime errors (logic flaws, FSM transition errors, wiring issues).
  • Diversity: Covers both Verilog and VHDL, ranging from simple syntax slips to complex logical "heisenbugs" like ring oscillator failures.

Key Insights and Ideas

  • Reasoning Models Rule: The "reasoning" class of models (o3-mini and DeepSeek R1) significantly outperformed standard chat models. Their ability to "think" through the logic of a circuit makes them much better at diagnosing runtime errors, which are inherently more abstract than syntax errors.
  • The RAG "Distraction" Effect: Interestingly, RAG was a double-edged sword for the older GPT-3.5-turbo. While RAG helped larger models, it actually degraded GPT-3.5's performance. The authors suggest that the extra context might have acted as "noise" that the smaller model couldn't filter effectively.
  • Synthesis is "Solved," Runtime is the Frontier: Almost all models handled synthesis errors well because they follow strict, documented rules. Runtime errors—where the code is syntactically correct but logically broken—remain the true test of an LLM's hardware "intuition."
  • Compliance vs. Helpfulness: When RAG was introduced, models often ignored the instruction "do not give code" and provided solutions anyway. While this technically violates the prompt, it suggests that the retrieved documents strongly nudged the models toward being "helpful" fixers rather than just explainers.

Key Results

  • Top Performance: OpenAI’s o3-mini achieved a perfect 100% Pass@5 score across all categories when using RAG.
  • RAG Gains: For GPT-4o and DeepSeek R1, RAG provided a noticeable boost in "Correct & Complete" scores for runtime errors, pushing them above 90%.
  • Consistency: The Avg@5 scores (which measure how often any given response is correct) showed that o3-mini and GPT-4o are highly reliable, consistently providing accurate explanations rather than just getting lucky once in five tries.

Limitations and Future Work

  • Dataset Scale: With only 34 samples, the benchmark is a great start but doesn't cover the vast "long tail" of complex industrial hardware bugs.
  • Manual Evaluation: The study relied on researchers manually grading responses. The authors acknowledge this isn't scalable and suggest that future work should explore "LLM-as-a-judge" for automated grading.
  • Single-Turn Interaction: The experiment used a single prompt. In the real world, debugging is a conversation. Future research could look at multi-turn "chat debugging" where the engineer and LLM iterate on a problem.
  • Security Implications: The authors note that many runtime errors are actually security vulnerabilities in disguise. Applying this RCA approach to hardware security is a promising next step.

Summary updated

Abstract

With advances in large language models (LLMs), new opportunities have emerged to develop tools that support the digital hardware design process. In this work, we explore how LLMs can assist with explaining the root cause of design issues and bugs that are revealed during synthesis and simulation, a necessary milestone on the pathway towards widespread use of LLMs in the hardware design process and for hardware security analysis. We find promising results: for our corpus of 34 different buggy scenarios, OpenAI's o3-mini reasoning model reached a correct determination 100% of the time under pass@5 scoring, with other state of the art models and configurations usually achieving more than 80% performance and more than 90% when assisted with retrieval-augmented generation.
BibTeX
@inproceedings{qiu2025towards,
  author = {Siyu Qiu and Muzhi Wang and Raheel Afsharmazayejani and Mohammad Moradi Shahmiri and Benjamin Tan and Hammond Pearce},
  title = {{Towards LLM-based Root Cause Analysis of Hardware Design Failures}},
  booktitle = {COINS},
  year = {2025},
  doi = {10.1109/coins65080.2025.11125748},
}

Loading related content...

Figure