Benchmarking Quantum Reinforcement Learning

Nico Meyer, Christian Ufrecht, George Yammine, Georgios Kontes, Christopher Mutschler, Daniel Scherer

Abstract

Researchers from Fraunhofer IIS have introduced a rigorous new framework for benchmarking Quantum Reinforcement Learning (QRL). By addressing the "reproducibility crisis" inherent in stochastic environments, they demonstrate that hybrid quantum models can be 10x more parameter-efficient than classical ones. However, they also reveal that claims of "quantum superiority" are often more nuanced than previously reported, depending heavily on the robustness of the classical baseline.

Addressing the Reproducibility Crisis

Reinforcement Learning (RL) is notoriously difficult to benchmark because performance often hinges on "random seeds"—the luck of the draw in model initialization or environment behavior. Quantum Reinforcement Learning (QRL) introduces additional layers of complexity, such as quantum "shot noise" and hardware imperfections.

Currently, the QRL field lacks a consensus on how to prove a quantum algorithm is genuinely superior. Many studies claim "quantum advantage" based on a handful of training runs on toy problems like CartPole. This paper addresses this instability by providing a mathematically grounded way to measure sample complexity—the number of interactions an agent needs with an environment before reaching a target performance level.

Comparison of performance metrics averaged over 5 runs versus 100 runs, highlighting how small sample sizes can lead to misleading conclusions about algorithm performance.

Methodology: A Robust Statistical Approach

The authors propose a two-pronged approach to standardize QRL benchmarking:

  1. The $\hat{S}$ Estimator: Instead of simple average reward curves, they define sample complexity as the number of steps required to reach a performance threshold with a specific probability ($\delta$). To ensure results are statistically significant, they mandate 100 independent training runs per configuration and use cluster re-sampling to calculate confidence intervals.
  2. Hybrid Classical-Quantum Architectures: The study focuses on Variational Quantum Circuits (VQCs). In these models, traditional neural network layers are replaced by quantum circuits where the "weights" are physical parameters (like microwave pulse angles) that can be optimized.

The BeamManagement6G Environment

To test these methods, the researchers developed BeamManagement6G, a realistic benchmark inspired by 6G wireless technology. In this environment, an agent must choose the optimal antenna beam to maintain signal intensity for a moving user. It is designed to be "classically hard" yet small enough for current quantum simulators to handle, offering a scalable difficulty level that moves beyond simple toy problems.

Key Insights and Results

10x Parameter Efficiency

One of the most striking findings is that a small hybrid quantum model with only 437 parameters (14 qubits) performed on par with a classical neural network containing 4,611 parameters. This suggests that quantum circuits may possess a superior "inductive bias" for certain spatial tasks, allowing them to learn complex patterns with significantly less "brain power."

Sample Complexity and Scaling

In the 6G environment, the 14-qubit hybrid model consistently outperformed similar-sized classical networks across various error thresholds. Interestingly, while classical models tend to "hit a wall" where adding more neurons yields diminishing returns, the quantum models tested did not yet show this saturation, hinting that scaling to more qubits might unlock performance levels unreachable by classical architectures.

Sample complexity $\hat{S}$ as a function of relative error $\epsilon$ and threshold probability $\delta$, comparing small classical, small quantum, and large classical models.

Limitations and the Path Forward

Despite the theoretical advantages, practical hurdles remain. Simulation costs are high; training a 14-qubit model for 100 epochs took roughly 12 minutes per run, compared to seconds for classical models. Furthermore, this study utilized noise-free simulations because current Noisy Intermediate-Scale Quantum (NISQ) hardware is not yet stable enough to support 100-run benchmarks reliably.

Future work will involve testing these benchmarks on actual quantum hardware and exploring whether the observed parameter efficiency holds as environments scale to multi-user 6G scenarios. The authors suggest that while a real-world "runtime" advantage may be years away, these benchmarks provide the necessary map to get there.

Summary updated

Abstract

Benchmarking and establishing proper statistical validation metrics for reinforcement learning (RL) remain ongoing challenges, where no consensus has been established yet. The emergence of quantum computing and its potential applications in quantum reinforcement learning (QRL) further complicate benchmarking efforts. To enable valid performance comparisons and to streamline current research in this area, we propose a novel benchmarking methodology, which is based on a statistical estimator for sample complexity and a definition of statistical outperformance. Furthermore, considering QRL, our methodology casts doubt on some previous claims regarding its superiority. We conducted experiments on a novel benchmarking environment with flexible levels of complexity. While we still identify possible advantages, our findings are more nuanced overall. We discuss the potential limitations of these results and explore their implications for empirical research on quantum advantage in QRL.
BibTeX
@inproceedings{meyer2025benchmarking,
  author = {Nico Meyer and Christian Ufrecht and George Yammine and Georgios Kontes and Christopher Mutschler and Daniel Scherer},
  title = {{Benchmarking Quantum Reinforcement Learning}},
  booktitle = {International Conference on Machine Learning},
  year = {2025},
  url = {https://openreview.net/forum?id=Ezp2elh9Yk},
}
Figure