Quantum Computer Benchmarking: An Explorative Systematic Literature Review

Tobias Rohe, Federico Harjes Ruiloba, Sabrina Egger, S. Beck, Jonas Stein, Claudia Linnhoff‐PopienarXiv 2025 · DOI · arXiv

Abstract

As quantum computing (QC) continues to evolve in hardware and software, measuring progress in this complex and diverse field remains a challenge. To track progress, uncover bottlenecks, and evaluate community efforts, benchmarks play a crucial role. But which benchmarking approach best addresses the diverse perspectives of QC stakeholders? We conducted the most comprehensive systematic literature review of this area to date, combining NLP-based clustering with expert analysis to develop a novel taxonomy and definitions for QC benchmarking, aligned with the quantum stack and its stakeholders. In addition to organizing benchmarks in distinct hardware, software, and application focused categories, our taxonomy hierarchically classifies benchmarking protocols in clearly defined subcategories. We develop standard terminology and map the interdependencies of benchmark categories to create a holistic, unified picture of the quantum benchmarking landscape. Our analysis reveals recurring design patterns, exposes research gaps, and clarifies how benchmarking methods serve different stakeholders. By structuring the field and providing a common language, our work offers a foundation for coherent benchmark development, fairer evaluation, and stronger cross-disciplinary collaboration in QC.

Introduction

As quantum computing transitions from theoretical physics to engineering reality, the industry faces a "Goldilocks" problem in performance measurement. Relying solely on qubit counts is too superficial, yet deep-dive physical characterizations are often too narrow for end-users. This fragmentation in measurement and terminology hinders the ability of stakeholders—from physicists building chips to industry experts solving optimization problems—to accurately track progress toward quantum advantage.

Methodology

This study provides a comprehensive synthesis of the benchmarking landscape through a Systematic Literature Review (SLR) of 329 relevant studies. The methodology employed a hybrid approach:

  1. NLP-Based Clustering: Using BERTopic, the authors generated document embeddings and used dimensionality reduction (UMAP) and clustering (HDBSCAN) to organize the literature.
  2. Expert Refinement: Initial clusters were labeled using GPT-4 and then manually refined by experts to align with the layers of the quantum stack.
  3. Interdependency Mapping: The authors analyzed how low-level hardware metrics (like gate fidelity) constrain and influence high-level application success.

A Unified Taxonomy for Quantum Benchmarking

The core contribution of this research is a stakeholder-centric taxonomy that organizes benchmarking into three primary perspectives: Hardware, Software, and Application focus. This taxonomy is built upon the foundational layers of the quantum computing stack.

The layers of the quantum computing stack, which serve as the structural basis for the proposed benchmarking taxonomy.

Hardware-Focused Benchmarking

Targeted at physicists and hardware engineers, this level focuses on physical characterization. Key metrics include $T_1$ and $T_2$ coherence times. The review highlights the trade-offs in tomography variants (State, Process, and Gate Set), noting that while Gate Set Tomography is highly accurate, its $O(d^4)$ scaling limits its utility for larger systems.

Software-Focused Benchmarking

This perspective serves algorithm developers and compiler designers by evaluating circuit-level performance and compiler efficiency. The analysis covers "Volumetric" benchmarks like IBM’s Quantum Volume and notes that newer protocols, such as Mirror Circuits, offer better scalability for the Noisy Intermediate-Scale Quantum (NISQ) era.

Application-Focused Benchmarking

Domain experts require end-to-end "time-to-solution" metrics for specific tasks in chemistry or finance. The study identifies Ground State Energy calculations and MaxCut optimization as the most mature application-focused benchmarks. A critical component here is "dequantization"—comparing quantum results against optimized classical baselines to ensure the reported quantum speedup is genuine.

Key Insights and Research Gaps

The review clarifies the distinction between benchmarking (measuring performance) and verification (checking correctness). A significant insight is the "cross-layer" effect: high-level application benchmarks often reveal system-wide issues, such as calibration drifts or crosstalk, that isolated low-level hardware tests may miss.

However, the authors identify a critical gap: there is a lack of "composite benchmarks" that explicitly link low-level physical metrics to high-level application performance. Furthermore, current research is largely restricted to universal, gate-based systems, leaving a need for expanded frameworks that include quantum annealers and analog simulators.

Conclusion

By structuring the field and providing a common language, this work offers a foundation for coherent benchmark development and fairer evaluation. Future work should focus on developing composite protocols that can predict application success based on hardware characterization and maintaining "living surveys" to keep pace with the rapid evolution of the quantum stack.

Summary updated

Abstract

As quantum computing (QC) continues to evolve in hardware and software, measuring progress in this complex and diverse field remains a challenge. To track progress, uncover bottlenecks, and evaluate community efforts, benchmarks play a crucial role. But which benchmarking approach best addresses the diverse perspectives of QC stakeholders? We conducted the most comprehensive systematic literature review of this area to date, combining NLP-based clustering with expert analysis to develop a novel taxonomy and definitions for QC benchmarking, aligned with the quantum stack and its stakeholders. In addition to organizing benchmarks in distinct hardware, software, and application focused categories, our taxonomy hierarchically classifies benchmarking protocols in clearly defined subcategories. We develop standard terminology and map the interdependencies of benchmark categories to create a holistic, unified picture of the quantum benchmarking landscape. Our analysis reveals recurring design patterns, exposes research gaps, and clarifies how benchmarking methods serve different stakeholders. By structuring the field and providing a common language, our work offers a foundation for coherent benchmark development, fairer evaluation, and stronger cross-disciplinary collaboration in QC.
BibTeX
@misc{rohe2025quantum,
  author = {Tobias Rohe and Federico Harjes Ruiloba and Sabrina Egger and S. Beck and Jonas Stein and Claudia Linnhoff‐Popien},
  title = {{Quantum Computer Benchmarking: An Explorative Systematic Literature Review}},
  year = {2025},
  eprint = {2509.03078},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.03078},
  howpublished = {arXiv},
}
Figure