LLM-aided explanations of EDA synthesis errors

Siyu Qiu, Benjamin Tan, Hammond Pearce

Abstract

Training new engineers in digital design is a challenge, particularly when it comes to teaching the complex electronic design automation (EDA) tooling used in this domain. Learners typically deploy designs in Verilog and VHDL to FPGAs via proprietary toolchains like Quartus Prime and Vivado. These tools are essential but notoriously difficult for novices to navigate. This work investigates if Large Language Models (LLMs) can generate novice-friendly explanations of compile-time synthesis error messages. By generating 936 error message explanations using three OpenAI LLMs over 21 different buggy code samples, we find that in approximately 71% of cases, LLMs provide correct and complete explanations suitable for novice learners.

Introduction

The steep learning curve of digital design is exacerbated by the cryptic nature of Electronic Design Automation (EDA) tools. Industry-standard suites like AMD’s Vivado or Intel’s Quartus Prime often produce synthesis errors that are vague or point to misleading lines of code. For a student, these "inscrutable walls of text" represent a significant barrier to entry, contributing to a global shortage of qualified chip designers. This research explores the potential of LLMs to act as pedagogical assistants, translating raw logs into clear, conceptual explanations without simply providing the solution.

Methodology

The study employed a structured "human-in-the-loop" evaluation framework to assess LLM performance across various dimensions of hardware design errors.

The experimental pipeline from bug creation to manual grading.

The researchers authored 21 representative synthesis-time bugs in both VHDL and Verilog, covering common mistakes like missing semicolons and type mismatches. These bugs were processed through Vivado and Quartus to harvest raw error logs. Three OpenAI models—GPT-3.5-turbo, GPT-4, and GPT-4-turbo-preview—were queried using two prompting strategies:

  1. Error & Code (E&C): Providing the error log and the entire source file.
  2. Error, Code, & Line (EC&L): Providing the log, the file, and highlighting the specific line flagged by the tool.

Responses were manually graded based on educational metrics, specifically focusing on "Concept Accuracy" and "Correct & Complete" explanations, while discouraging the provision of direct code solutions to maintain a constructivist pedagogical approach.

Key Results and Analysis

The evaluation of 936 responses revealed that LLMs are highly effective at identifying the underlying hardware concepts, even when the specific explanation might be incomplete.

Summary of performance metrics across different experimental variables.

Prompting and Tool Influence

Contextual information proved vital. The EC&L strategy, which included the specific line number flagged by the EDA tool, increased the "Correct & Complete" rate from 67.31% to 75.21%. Furthermore, the quality of the source tool mattered; Quartus logs, which are generally more descriptive, led to better LLM explanations compared to Vivado logs.

A notable finding was the "Small Model Paradox": GPT-3.5-turbo achieved a higher "Correct & Complete" rate (74.36% at pass@10) than the larger GPT-4 models. However, GPT-4 variants were superior at avoiding factual inaccuracies and maintaining conceptual accuracy.

Language-wise, Verilog errors were explained more accurately than VHDL errors (77.45% vs 66.48% "Correct & Complete"). This likely reflects the higher prevalence of Verilog in the open-source datasets used to train these models.

Discussion and Limitations

While LLMs show great promise for synthesis-time bugs, this study did not address run-time or logic errors. Logic errors, which require the analysis of simulation waveforms, present a significantly higher complexity for current text-based LLMs. Additionally, a challenge for pedagogical use is the "over-help" problem: more capable models often provided direct code fixes despite instructions to the contrary, which may bypass the student's learning process.

Conclusion

This research demonstrates that LLMs can successfully bridge the gap between complex EDA tool output and student understanding. With an overall success rate of over 71%, these models could be integrated directly into IDEs to provide real-time, conversational debugging support, potentially lowering the barrier to entry for the next generation of digital designers.

Summary updated

Abstract

Training new engineers in digital design is a challenge, particularly when it comes to teaching the complex electronic design automation (EDA) tooling used in this domain. Learners will typically deploy designs in the Verilog and VHDL hardware description languages to Field Programmable Gate Arrays (FPGAs) from Altera (Intel) and Xilinx (AMD) via proprietary closed-source toolchains (Quartus Prime and Vivado, respectively). These tools are complex and difficult to use -- yet, as they are the tools used in industry, they are an essential first step in this space. In this work, we examine how recent advances in artificial intelligence may be leveraged to address aspects of this challenge. Specifically, we investigate if Large Language Models (LLMs), which have demonstrated text comprehension and question-answering capabilities, can be used to generate novice-friendly explanations of compile-time synthesis error messages from Quartus Prime and Vivado. To perform this study we generate 936 error message explanations using three OpenAI LLMs over 21 different buggy code samples. These are then graded for relevance and correctness, and we find that in approximately 71% of cases the LLMs give correct & complete explanations suitable for novice learners.
BibTeX
@misc{qiu2024llm,
  author = {Siyu Qiu and Benjamin Tan and Hammond Pearce},
  title = {{LLM-aided explanations of EDA synthesis errors}},
  year = {2024},
  eprint = {2404.07235},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2404.07235},
}
Figure