Learning to Compare Hardware Designs for High-Level Synthesis

Yunsheng Bai, Atefeh Sohrabizadeh, Zijian Ding, Rongjian Liang, Weikai Li, Ding Wang, Haoxing Ren, Yizhou Sun, Jason Cong

TL;DR

compareXplore is a new machine learning framework that optimizes hardware design by teaching models how to compare two designs rather than just predicting the performance of one in isolation. By using a novel "Node Difference Attention" mechanism and a two-stage exploration strategy, it achieves a 16.11% average reduction in hardware latency compared to current state-of-the-art methods.

What Problem Does This Paper Address?

In High-Level Synthesis (HLS), engineers transform high-level code (like C++) into hardware descriptions (like RTL). To optimize this, they use pragmas—directives that tell the compiler how to handle loops, arrays, and parallelization.

The challenge is that the "design space" is massive and highly non-linear. Changing one pragma might have a ripple effect on others, making it incredibly difficult to predict exactly how fast a design will be. Existing machine learning approaches (like the HARP model) try to predict absolute performance values (pointwise prediction). However, the authors argue that predicting an exact latency value is unnecessarily hard; for optimization, we really just need to know if Design A is better than Design B.

Methods

The authors introduce compareXplore, which shifts the focus to comparative learning. The architecture consists of three main innovations:

  1. Node Difference Attention: When comparing two designs, the model doesn't just look at them separately. This module specifically calculates the differences between the Graph Neural Network (GNN) embeddings of two designs. It uses an attention mechanism to "zoom in" on the specific pragma changes that most significantly impact performance.
  2. Hybrid Loss Function: The model is trained using a combination of two losses:
  3. Pointwise Loss (MSE): To keep a sense of absolute performance.
  4. Pairwise Loss (Cross-Entropy): To master the art of ranking one design over another.
  5. Two-Stage Design Space Exploration (DSE):
    • Stage 1 (Pruning): A fast pointwise model quickly discards obviously bad designs, narrowing the field to the top $K_1$ candidates.
  6. Stage 2 (Verification): The pairwise comparison model performs a "tournament" among the remaining candidates. Inspired by Ranked Choice Voting (RCV), the model iteratively eliminates the "least preferred" designs until the best ones remain.

Datasets

The researchers evaluated their approach using a dataset of 40 kernels (computational tasks) from various domains like image processing and signal processing.

  • Total Designs: 10,868 unique hardware designs.
  • Tooling: Designs were synthesized using Xilinx Vitis HLS (v2021.1).
  • Split: Roughly 90% for training, with the remainder split between validation and testing.

Key Insights and Ideas

  • Comparison is Easier than Prediction: The model actually had a higher regression error (RMSE) than the baseline HARP, yet it found better hardware designs. This confirms the "aha moment": a model doesn't need to be a perfect calculator to be a great scout.
  • Focusing on Delta: By explicitly modeling the difference between designs (via Node Difference Attention), the model learns to ignore the parts of the code that stay the same and focus entirely on the impact of specific pragma tweaks.
  • The "Sweet Spot" for $\alpha$: The authors found that a balanced weight ($\alpha=1$) between the pointwise and pairwise loss functions yielded the best ranking performance (Kendall’s $\tau$). Too much emphasis on one or the other degraded the results.

Key Results

  • Latency Reduction: compareXplore achieved a 16.11% average reduction in latency across all tested kernels compared to HARP.
  • Extreme Wins: In specific cases, like the "adi" kernel, the tool reduced latency by nearly 50%.
  • Ranking Accuracy: The model showed significantly better alignment with true performance rankings, achieving a higher Kendall’s $\tau$ score (0.4319 vs 0.4157).
  • Efficiency: Despite the complexity of pairwise comparisons, Stage 2 only accounted for about 12.94% of the total computation time, making the approach practical for real-world engineering workflows.

Limitations and Future Work

  • Regression Accuracy: As noted, the model is less accurate at predicting the exact latency value compared to pointwise-only models. While this doesn't hurt its ability to find the best design, it might be a drawback if an engineer needs a precise performance estimate without running the full HLS tool.
  • Hard Comparisons: The model still finds it difficult to distinguish between designs with very small pragma differences (the $d_1$ category in their tests). The authors suggest curriculum learning—starting with easy comparisons and moving to harder ones—as a potential fix.
  • LLM Integration: The authors propose that future versions could integrate with Large Language Models (LLMs) to allow for a more intuitive, natural-language-driven hardware design process.

Summary updated

Abstract

High-level synthesis (HLS) is an automated design process that transforms high-level code into hardware designs, enabling the rapid development of hardware accelerators. HLS relies on pragmas, which are directives inserted into the source code to guide the synthesis process, and pragmas have various settings and values that significantly impact the resulting hardware design. State-of-the-art ML-based HLS methods, such as HARP, first train a deep learning model, typically based on graph neural networks (GNNs) applied to graph-based representations of the source code and pragmas. They then perform design space exploration (DSE) to explore the pragma design space, rank candidate designs using the model, and return the top designs. However, traditional DSE methods face challenges due to the highly nonlinear relationship between pragma settings and performance metrics, along with complex interactions between pragmas that affect performance in non-obvious ways. To address these challenges, we propose compareXplore, a novel approach that learns to compare hardware designs for effective HLS optimization. CompareXplore introduces a hybrid loss function that combines pairwise preference learning with pointwise performance prediction, enabling the model to capture both relative preferences and absolute performance. Moreover, we introduce a novel node difference attention module that focuses on the most informative differences between designs, enabling the model to identify critical pragmas impacting performance. CompareXplore adopts a two-stage DSE, where a pointwise prediction model is used for the initial design pruning, followed by a pairwise comparison stage for precise performance verification. In extensive experiments, compareXplore achieves significant improvements in ranking metrics and generates high-quality HLS results for the selected designs, outperforming the existing SOTA method.
BibTeX
@inproceedings{bai2024learning,
  author = {Yunsheng Bai and Atefeh Sohrabizadeh and Zijian Ding and Rongjian Liang and Weikai Li and Ding Wang and Haoxing Ren and Yizhou Sun and Jason Cong},
  title = {{Learning to Compare Hardware Designs for High-Level Synthesis}},
  booktitle = {MLCAD},
  year = {2024},
  doi = {10.1145/3670474.3685940},
}

Loading related content...

Figure