GenBen:A Genarative Benchmark for LLM-Aided Design
Lists
Only you see this
Abstract
This paper introduces GenBen, a generative benchmark designed to evaluate the capabilities of large language models (LLMs) in hardware design. With the rapid advancement of LLM-aided design (LAD), it has become crucial to assess the effectiveness of these models in automating hardware design processes. Existing benchmarks primarily focus on hardware code generation and often neglect critical aspects such as Quality-of-Result (QoR) metrics, design diversity, modality, and test set contamination. GenBen is the first open-source, generative benchmark tailored for LAD that encompasses a range of tasks, from high-level architecture to low-level circuit optimization, and includes diverse, silicon-proven hardware designs. We have also designed a difficulty tiering mechanism to provide fine-grained insights into enhancements of LLM-aided designs. Through extensive evaluations of several state-of-the-art LLMs using GenBen, we reveal their strengths and weaknesses in hardware design automation. Our findings are based on 10,920 experiments and 2,160 hours of evaluation, underscoring the potential of this work to significantly advance the LAD research community. In addition, both GenBen employs an end-to-end testing infrastructure to ensure consistent and reproducible results across different LLMs. The benchmark is available at https://anonymous.4open.science/r/GENBEN-2812.
BibTeX
@misc{wan2025genben,
author = {Gwok-Waa Wan and Wang yubo and SamZaak Wong and jingyi zhang and Mengnv Xing and Zhe jiang and Nan Guan and ying wang and Ning Xu and Qiang Xu and Xi Wang},
title = {{GenBen:A Genarative Benchmark for LLM-Aided Design}},
year = {2025},
url = {https://openreview.net/forum?id=gtVo4xcpFI},
howpublished = {OpenReview},
note = {Submitted to ICLR 2025},
}Loading related content...