Exploring Code Language Models for Automated HLS-based Hardware Generation: Benchmark, Infrastructure and Analysis

Jiahao Gai, Hao Mark Chen, Zhican Wang, Hongyu Zhou, Wanru Zhao, Nicholas Lane, Hongxiang Fan

TL;DR

Using Large Language Models (LLMs) to generate low-level hardware description languages like Verilog is notoriously difficult due to severe data scarcity and high token costs. This paper proposes shifting the target to High-Level Synthesis (HLS), introducing a massive dataset of 42,000+ HLS programs alongside a framework that combines fine-tuning, Chain-of-Thought prompting, and automated feedback loops to achieve highly accurate hardware generation.

What Problem Does This Paper Address?

While LLMs excel at generating software code (like Python or C++), applying them to hardware generation faces major roadblocks. Traditional approaches focus on low-level Hardware Description Languages (HDLs) like Verilog or SystemVerilog. However, HDLs suffer from three distinct challenges:

  1. Data Scarcity: HDL code makes up less than 1% of standard open-source code datasets compared to software languages.
  2. The Knowledge Gap: Pre-trained LLMs are optimized for software semantics, making their knowledge poorly transferable to the concurrent, hardware-specific nature of HDLs.
  3. Token Inefficiency: Implementing logic in HDL is highly verbose, requiring 3 to 4 times more tokens than software, which drives up LLM inference costs and latency.
Comparison of data availability between HDLs and software languages

To solve this, the authors explore leveraging LLMs to generate High-Level Synthesis (HLS) code instead. HLS allows developers to describe hardware using C/C++ syntax, directly inheriting the vast software knowledge already embedded within pre-trained LLMs.

Token usage comparison between HLS and Verilog

Methods

The authors introduce an end-to-end framework divided into two primary stages: model fine-tuning and iterative code generation.

Overview of the proposed automated hardware generation framework
  • Supervised Fine-Tuning (SFT): They fine-tune a Code-Llama-7B model using the low-rank adaptation (QLoRA) technique on a newly compiled HLS dataset.
  • Chain-of-Thought (CoT) Prompting: The framework injects a structured hardware engineering checklist into the prompt (e.g., considering FPGA characteristics, defining program structure, writing logic, and selecting data types) to guide the model's reasoning step-by-step.
  • Two-Step Feedback Loops: During inference, the generated HLS code is automatically evaluated. First, a syntax check is performed via the GCC compiler using the -fsyntax-only flag (enabling rapid verification without full compilation overhead). Second, if the syntax passes, the code runs through functional unit tests. Any discovered errors or defects are fed back into the prompt for iterative self-correction.

Datasets

The paper introduces a substantial benchmark dataset built by collecting 52 open-source HLS designs spanning matrix operations, scientific simulations, and iterative methods. By applying various hardware optimization pragmas (such as PIPELINE, PARALLEL, and TILE), they expanded these base designs into a dataset of over 42,000 valid HLS programs.

To create natural language prompts for training, they used ChatGPT (versions 3.5 and 4) to generate design descriptions. For evaluation, they created two test sets:

  • MachineGen: Raw descriptions generated entirely by the LLM.
  • HumanRefine: Manually polished, concise, human-like instructions.

Key Insights and Ideas

  • The HLS Advantage: Shifting from Verilog to HLS fundamentally bypasses the data scarcity problem. Because HLS relies on C/C++ syntax, the model can effectively reuse its extensive software pre-training.
  • The Prompt Disparity: Surprisingly, the model performed significantly worse on human-refined prompts (47.29% syntax correctness) than on machine-generated ones (93.83%). This reveals a strong training bias toward explicit, highly detailed machine-generated text; human prompts often omit repetitive details that LLMs rely on for context.
  • Hardware CoT is Linear: While general software generation often requires complex, specialized CoT frameworks, hardware generation benefits immensely from a straightforward, linear engineering checklist built directly into the prompt.

Key Results

The combination of fine-tuning, CoT, and feedback loops yielded dramatic performance improvements:

  • Fine-Tuning Impact: The base, non-fine-tuned model failed 100% of functional tests. Fine-tuning skyrocketed syntax correctness from 54.85% to 88.44% and functional pass rates to 53.20%.
  • CoT Boost: Adding Chain-of-Thought prompting further raised syntax correctness to 94.33% and functional correctness to 61.45%.
  • Feedback Loops: Iterative debugging loops successfully corrected complex code blocks, though the gains showed diminishing returns after the first loop iteration.
Quantitative impact of fine-tuning and CoT prompting

The generated HLS designs were successfully synthesized using Xilinx Vivado targeting a Xilinx VCU118 FPGA, demonstrating realistic hardware resource utilization and performance.

Limitations and Future Work

While promising, the authors note several limitations:

  • Model Constraints: The evaluation was limited to Code-Llama-7B. Advanced reasoning models (like DeepSeek-R1) and test-time scaling approaches were not explored.
  • Dataset Diversity: Although the dataset contains 42,000 programs, they are variations derived from only 52 unique base designs, limiting structural diversity.
  • Missing PPA Feedback: The feedback loop currently only checks for syntax and functional correctness. It does not yet optimize for hardware performance metrics like Power, Performance, and Area (PPA).

Summary updated

Abstract

Recent advances in code generation have illuminated the potential of employing large language models (LLMs) for general-purpose programming languages such as Python and C++, opening new opportunities for automating software development and enhancing programmer productivity. The potential of LLMs in software programming has sparked significant interest in exploring automated hardware generation and automation. Although preliminary endeavors have been made to adopt LLMs in generating hardware description languages (HDLs), several challenges persist in this direction. First, the volume of available HDL training data is substantially smaller compared to that for software programming languages. Second, the pre-trained LLMs, mainly tailored for software code, tend to produce HDL designs that are more error-prone. Third, the generation of HDL requires a significantly higher number of tokens compared to software programming, leading to inefficiencies in cost and energy consumption. To tackle these challenges, this paper explores leveraging LLMs to generate High-Level Synthesis (HLS)-based hardware design. Although code generation for domain-specific programming languages is not new in the literature, we aim to provide experimental results, insights, benchmarks, and evaluation infrastructure to investigate the suitability of HLS over low-level HDLs for LLM-assisted hardware design generation. To achieve this, we first finetune pre-trained models for HLS-based hardware generation, using a collected dataset with text prompts and corresponding reference HLS designs. An LLM-assisted framework is then proposed to automate end-to-end hardware code generation, which also investigates the impact of chain-of-thought and feedback loops promoting techniques on HLS-design generation. Limited by the timeframe of this research, we plan to evaluate more advanced reasoning models in the future.
BibTeX
@inproceedings{gai2025exploring,
  author = {Jiahao Gai and Hao Mark Chen and Zhican Wang and Hongyu Zhou and Wanru Zhao and Nicholas Lane and Hongxiang Fan},
  title = {{Exploring Code Language Models for Automated HLS-based Hardware Generation: Benchmark, Infrastructure and Analysis}},
  booktitle = {ASP-DAC},
  year = {2025},
}
Figure