Towards LLM-based Root Cause Analysis of Hardware Design Failures

Siyu Qiu, Muzhi Wang, Raheel Afsharmazayejani, Mohammad Moradi Shahmiri, Benjamin Tan, Hammond Pearce

Abstract

With advances in large large models (LLMs), new opportunities have emerged to develop tools that support the digital hardware design process. In this work, we explore how LLMs can assist with explaining the root cause of design issues and bugs that are revealed during synthesis and simulation, a necessary milestone on the pathway towards widespread use of LLMs in the hardware design process and for hardware security analysis. We find promising results: for our corpus of 34 different buggy scenarios, OpenAI's o3-mini reasoning model reached a correct determination 100% of the time under pass@5 scoring, with other state of the art models and configurations usually achieving more than 80% performance and more than 90% when assisted with retrieval-augmented generation.

Introduction

In the digital hardware design lifecycle, debugging remains a notorious bottleneck. When a design fails during synthesis—the process of translating code into hardware gates—or simulation, engineers are often met with cryptic error messages. Understanding the "why" behind a failure requires deep domain knowledge of Hardware Description Languages (HDLs) like Verilog and VHDL, as well as the specific quirks of Electronic Design Automation (EDA) tools.

While previous research has focused on using LLMs to generate hardware code, this work shifts the focus to Root Cause Analysis (RCA). The goal is to support a "human-in-the-loop" workflow where the LLM explains the bug to the engineer, effectively acting as a tutor or a senior reviewer to speed up the repair process and upskill the designer.

Methodology

The researchers evaluated four state-of-the-art models: GPT-3.5-turbo, GPT-4o, o3-mini, and DeepSeek R1. They tested two primary prompting strategies: a baseline approach providing only the buggy code and error message, and a Retrieval-Augmented Generation (RAG) approach.

Overview of the RAG-based Root Cause Analysis workflow, showing the path from buggy code to LLM evaluation.

The RAG system augments prompts with relevant snippets from a custom database of hardware textbooks, tutorials, and tool manuals. To evaluate the quality of the explanations, outputs were manually graded based on concept accuracy, factuality, relevance, and whether the answer was correct and complete.

The root-cause-bench Dataset

To rigorously test these models, the researchers introduced a new, open-source benchmark: root-cause-bench. It consists of 34 manually labeled hardware design errors categorized into:

  • Synthesis-time errors (19): Including syntax errors, type mismatches, and illegal assignments.
  • Simulation runtime errors (15): Including logic flaws, Finite State Machine (FSM) transition errors, and wiring issues.

The dataset covers both Verilog and VHDL, ranging from simple syntax slips to complex logical "heisenbugs" like ring oscillator failures.

Key Results

The evaluation revealed that "reasoning" class models (o3-mini and DeepSeek R1) significantly outperform standard chat models. Their ability to chain logic makes them particularly adept at diagnosing runtime errors, which are inherently more abstract than syntax errors.

Pass@5 scores across models, highlighting o3-mini's perfect performance and the impact of RAG.

Key findings include:

  • Top Performance: OpenAI’s o3-mini achieved a perfect 100% Pass@5 score across all categories when using RAG.
  • RAG Gains: For GPT-4o and DeepSeek R1, RAG provided a noticeable boost in "Correct & Complete" scores for runtime errors, pushing them above 90%.
  • The RAG "Distraction" Effect: Interestingly, RAG was a double-edged sword for the older GPT-3.5-turbo. While it helped larger models, the extra context acted as "noise" that degraded the performance of the smaller model.

Conclusion and Future Work

While synthesis errors are largely "solved" by these models due to their adherence to strict rules, runtime errors remain the true frontier for LLM hardware intuition. Future work will focus on scaling the benchmark to cover industrial-scale bugs, exploring automated "LLM-as-a-judge" grading, and investigating multi-turn interactions for interactive debugging. Furthermore, applying this RCA approach to hardware security vulnerabilities remains a promising avenue for future research.

Summary updated

Abstract

With advances in large language models (LLMs), new opportunities have emerged to develop tools that support the digital hardware design process. In this work, we explore how LLMs can assist with explaining the root cause of design issues and bugs that are revealed during synthesis and simulation, a necessary milestone on the pathway towards widespread use of LLMs in the hardware design process and for hardware security analysis. We find promising results: for our corpus of 34 different buggy scenarios, OpenAI's o3-mini reasoning model reached a correct determination 100% of the time under pass@5 scoring, with other state of the art models and configurations usually achieving more than 80% performance and more than 90% when assisted with retrieval-augmented generation.
BibTeX
@inproceedings{qiu2025towards,
  author = {Siyu Qiu and Muzhi Wang and Raheel Afsharmazayejani and Mohammad Moradi Shahmiri and Benjamin Tan and Hammond Pearce},
  title = {{Towards LLM-based Root Cause Analysis of Hardware Design Failures}},
  booktitle = {COINS},
  year = {2025},
  doi = {10.1109/coins65080.2025.11125748},
}
Figure