QuanBench: Benchmarking Quantum Code Generation with Large Language Models

Xiaoyu Guo, M Wang, Jianjun ZhaoarXiv 2025 · DOI · arXiv

TL;DR

QuanBench is a new benchmark designed to evaluate how well Large Language Models (LLMs) generate quantum computing code. While LLMs excel at classical programming, this study reveals they struggle significantly with quantum logic, with even the best models failing to exceed a 40% success rate on first-attempt accuracy.

What Problem Does This Paper Address?

Large Language Models have become remarkably proficient at writing Python or C++, but quantum programming is a different beast entirely. Unlike classical code, quantum programs operate on qubits, utilize unitary transformations, and rely heavily on complex linear algebra and probabilistic outcomes.

The authors identify a major gap: existing benchmarks (like Qiskit HumanEval) focus primarily on whether a model can use the right API calls. They don't adequately test if the model understands the underlying "quantumness"—the algorithmic reasoning required to prepare entangled states, decompose gates, or implement complex algorithms like Grover’s or Shor’s. There is a lack of a systematic way to measure both the functional correctness (does it run?) and semantic equivalence (is the resulting quantum circuit mathematically what we asked for?) of LLM-generated quantum code.

Methods

The researchers developed QuanBench, a suite of 44 curated quantum programming tasks (currently expanding to 117). To evaluate the models, they moved beyond simple "pass/fail" unit tests and implemented a dual-metric system:

  1. Pass@K: A standard metric measuring the probability that at least one of the top $K$ generated code samples is correct.
  2. Process Fidelity: This is the "secret sauce." It compares the unitary matrix of the LLM-generated circuit to a canonical reference solution. A fidelity of 1.0 means the circuits are mathematically identical, even if the code looks different.
  3. KL Divergence: For tasks involving measurement (like Bell state preparation), they compare the probability distribution of the model's output to the expected distribution.

The evaluation covered nine state-of-the-art models, including general-purpose giants like GPT-4.1 and Claude 3.7, alongside code-specialized models like DeepSeek R1 and CodeLlama.

Datasets

The authors curated the QuanBench dataset by:

  • Sourcing high-quality quantum implementations from GitHub and StackOverflow.
  • Filtering for repositories created after 2023 to minimize "data leakage" (ensuring the models hadn't already seen the code during their initial training).
  • Categorizing tasks into four domains: Quantum Algorithms (e.g., Grover, QFT), Quantum State Preparation (e.g., GHZ states), Gate Decomposition, and Quantum Machine Learning.

Key Insights and Ideas

  • The "Surface-Level" Trap: Models often capture the "surface" of a problem (using the right library names) but fail on the "deep" logic. For example, a model might generate a circuit that passes a basic test but has a low Process Fidelity because it added unnecessary gates or messed up the phase.
  • Stale Knowledge: A recurring failure was the use of deprecated APIs. Many models insisted on using the cu1 gate, which was replaced by cp in Qiskit versions after 0.45.0. This suggests LLM training data is lagging behind the rapid evolution of quantum software frameworks.
  • Reasoning vs. Scale: Interestingly, model performance didn't strictly correlate with size. Domain-specific adaptation and "reasoning" capabilities (seen in models like DeepSeek R1) proved more important than raw parameter count.
  • Structural Sensitivity: LLMs frequently struggle with qubit indexing—assigning the same qubit as both the control and target in a gate, which is a fundamental "syntax" error in quantum circuit construction.

Key Results

  • Low Overall Accuracy: No model achieved a Pass@1 accuracy higher than 40%. Even with five attempts (Pass@5), the best score topped out at 50%.
  • Top Performers: DeepSeek R1 and Gemini 2.5 Flash emerged as the leaders, solving 22 out of 44 tasks. DeepSeek R1 was particularly strong in quantum state preparation.
  • The Fidelity Gap: Even when models produced "correct" code that passed unit tests, the average Process Fidelity was only around 51.8%. This indicates that LLMs often produce "noisy" or inefficient circuits to reach the right answer.
  • Specialized Models Lagging: Surprisingly, some code-specialized models like CodeLlama performed significantly worse than general-purpose models, suggesting they haven't been exposed to enough quantum-specific data.

Limitations and Future Work

The authors acknowledge several areas for growth:

  • Framework Diversity: Currently, QuanBench is limited to the Qiskit framework. Future versions aim to include Cirq, PennyLane, and others.
  • Task Complexity: The benchmark currently focuses on algorithm-level tasks. It does not yet cover low-level hardware concerns like pulse-level control or error mitigation.
  • QML Scarcity: The Quantum Machine Learning category is currently small due to a lack of standardized, publicly available implementations to draw from.
  • Prompt Engineering: The study used a standardized prompt; exploring how different prompting strategies (like Chain-of-Thought) might improve quantum reasoning is a logical next step.

Summary updated

Abstract

Large language models (LLMs) have demonstrated good performance in general code generation; however, their capabilities in quantum code generation remain insufficiently studied. This paper presents QuanBench, a benchmark for evaluating LLMs on quantum code generation. QuanBench includes 44 programming tasks that cover quantum algorithms, state preparation, gate decomposition, and quantum machine learning. Each task has an executable canonical solution and is evaluated by functional correctness (Pass@K) and quantum semantic equivalence (Process Fidelity). We evaluate several recent LLMs, including general-purpose and code-specialized models. The results show that current LLMs have limited capability in generating the correct quantum code, with overall accuracy below 40% and frequent semantic errors. We also analyze common failure cases, such as outdated API usage, circuit construction errors, and incorrect algorithm logic. QuanBench provides a basis for future work on improving quantum code generation with LLMs.
BibTeX
@misc{guo2025quanbench,
  author = {Xiaoyu Guo and M Wang and Jianjun Zhao},
  title = {{QuanBench: Benchmarking Quantum Code Generation with Large Language Models}},
  year = {2025},
  eprint = {2510.16779},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.16779},
  howpublished = {arXiv},
}
Figure