Abstract
Training new engineers in digital design is a challenge, particularly when it comes to teaching the complex electronic design automation (EDA) tooling used in this domain. Learners will typically deploy designs in the Verilog and VHDL hardware description languages to Field Programmable Gate Arrays (FPGAs) from Altera (Intel) and Xilinx (AMD) via proprietary closed-source toolchains (Quartus Prime and Vivado, respectively). These tools are complex and difficult to use -- yet, as they are the tools used in industry, they are an essential first step in this space. In this work, we examine how recent advances in artificial intelligence may be leveraged to address aspects of this challenge. Specifically, we investigate if Large Language Models (LLMs), which have demonstrated text comprehension and question-answering capabilities, can be used to generate novice-friendly explanations of compile-time synthesis error messages from Quartus Prime and Vivado. To perform this study we generate 936 error message explanations using three OpenAI LLMs over 21 different buggy code samples. These are then graded for relevance and correctness, and we find that in approximately 71% of cases the LLMs give correct & complete explanations suitable for novice learners.
The Challenge of EDA Tooling
Electronic Design Automation (EDA) tools like Intel’s Quartus Prime and AMD’s Vivado are industrial powerhouses, but they are notoriously unfriendly to beginners. When a student writes buggy VHDL or Verilog code, the compiler often throws an error that is either vague or misleading—sometimes pointing to the wrong line of code entirely. This creates a steep learning curve that contributes to a global shortage of qualified chip designers. This research investigates whether LLMs can bridge this gap by taking a confusing raw error message and the buggy source code to produce a "novice-friendly" explanation that helps the student learn why their code failed.
Methodology
The researchers developed a proof-of-concept pipeline to test three OpenAI models: gpt-3.5-turbo, gpt-4, and gpt-4-turbo-preview. The methodology involved two types of prompt engineering: one providing the error message and full code (E&C), and another adding the specific line number reported by the tool (EC&L).
Grading was based on pedagogical principles of constructivism, evaluating 936 generated explanations on four primary metrics:
- Concept Accurate: Linking to correct hardware concepts.
- No Inaccuracies: Factual correctness of the information.
- Correct & Complete: Providing all necessary information to fix the bug.
- Solution Provided: Whether the model "over-helped" by providing the direct answer (considered a negative for learning purposes).
Experimental Dataset
The study introduced a new, open-source dataset of 21 representative synthesis-time bugs. These cover 10 distinct categories common among novices, such as syntax errors, width mismatches, multiple driver errors, and type conversion issues. The dataset includes both VHDL and Verilog examples, with error messages harvested directly from industrial logs of Quartus Prime and Vivado.
Results and Analysis
Overall, LLMs provided "correct and complete" explanations in 71.26% of cases. In 94.23% of cases, the LLMs correctly identified the underlying hardware concepts involved in the error. One of the most significant findings for education was that the models only provided a direct "copy-paste" solution 3.31% of the time, forcing students to engage with the explanation.
Key Insights
- The "Verilog Advantage": LLMs were generally better at explaining Verilog errors than VHDL errors, likely due to the relative abundance of Verilog in open-source training data.
- Contextual Anchors: Providing the specific line number (EC&L prompt) improved the "Correct & Complete" rate from 67.31% to 75.21%. Even if the tool points to the wrong line, the extra anchor helps the LLM reason about the code structure.
- Tool-Specific Performance: LLMs performed better with Quartus errors (72.71% correct & complete) than Vivado (69.74%). Analysis suggests Quartus’s raw error messages contain more descriptive keywords compared to Vivado’s more generic syntax flags.
- Model Comparison: Surprisingly, the smaller GPT-3.5 model outperformed the larger GPT-4 models in the "Correct & Complete" metric (69.23% vs 53.85% for GPT-4 pass@1), though GPT-4 was superior at avoiding factual inaccuracies.
Conclusion and Future Work
This study demonstrates that LLMs can effectively act as pedagogical assistants for hardware design. While highly successful for synthesis-time errors, future work is needed to address run-time or logic errors, which require analyzing simulation waveforms (like VCD files). Integrating these explanations directly into IDEs could provide real-time, novice-friendly feedback to the next generation of hardware engineers.
Summary updated