Benchmarking Quantum Reinforcement Learning

Nico Meyer, Christian Ufrecht, George Yammine, Georgios D. Kontes, Christopher Mutschler, Daniel D. Scherer

TL;DR

Researchers from Fraunhofer IIS have introduced a rigorous new framework for benchmarking Quantum Reinforcement Learning (QRL). By moving beyond "lucky" single-run comparisons and introducing a scalable, 6G-inspired environment, they demonstrate that while hybrid quantum models can be 10x more parameter-efficient than classical ones, claims of "quantum superiority" are often more nuanced than previously reported.

What Problem Does This Paper Address?

Reinforcement Learning (RL) is notoriously difficult to benchmark. Performance often hinges on "random seeds"—the luck of the draw in how a model is initialized or how the environment behaves. Quantum Reinforcement Learning (QRL) adds even more layers of chaos, including quantum "shot noise" and hardware imperfections.

Currently, the QRL field lacks a consensus on how to prove a quantum algorithm is actually better. Many papers claim "quantum advantage" based on a handful of training runs or toy problems like CartPole. This paper addresses this "reproducibility crisis" by providing a mathematically grounded way to measure sample complexity—essentially, how many interactions with an environment an agent needs before it "gets it."

Methods

The authors propose a two-pronged approach to clean up QRL benchmarking:

  1. A Robust Statistical Estimator ($\hat{S}$): Instead of just looking at average reward curves, they define sample complexity as the number of steps required to reach a performance threshold with a specific probability ($\delta$). To ensure results aren't flukes, they mandate a massive increase in scale: performing 100 independent training runs per configuration and using "cluster re-sampling" to calculate confidence intervals.
  2. Hybrid Classical-Quantum Architectures: They focus on "Variational Quantum Circuits" (VQCs). In these models, the heavy lifting of a traditional neural network is replaced by a quantum circuit where the "weights" are physical parameters (like the angle of a microwave pulse) that can be optimized.

Datasets

The paper utilizes two main environments:

  • BeamManagement6G (New): A novel, realistic benchmark inspired by 6G wireless tech. The agent must choose the best antenna to maintain signal intensity for a moving user. It’s designed to be "classically hard" but small enough for current quantum simulators to handle, and it can be scaled in difficulty.
  • CartPole-v1: The "Hello World" of RL, used here to show how traditional benchmarking can lead to misleading conclusions about quantum performance.

Key Insights and Ideas

  • The 10x Parameter Efficiency: One of the most striking findings is that a small hybrid quantum model with only 437 parameters (14 qubits) performed on par with a classical neural network containing 4,611 parameters. This suggests that quantum circuits might have a superior "inductive bias" for certain spatial tasks, allowing them to learn complex patterns with far less "brain power."
  • The "Nuanced" Advantage: When compared to classical models of the same size, the quantum models won handily. However, when compared to the best possible classical model, the gap narrowed. The authors warn that "quantum advantage" is often a moving target that depends heavily on how much effort you put into tuning the classical baseline.
  • Saturation Sensitivity: Classical models tend to "hit a wall" where adding more neurons doesn't help. Interestingly, the quantum models tested (up to 14 qubits) didn't show this same saturation yet, hinting that scaling to more qubits might unlock even higher performance levels that classical models can't reach.

Key Results

  • Sample Complexity: In the 6G environment, the 14-qubit hybrid model consistently outperformed similar-sized classical networks across various error thresholds.
  • Reliability: By running 100 seeds, the authors showed that "Algorithm A" might look better than "Algorithm B" on average over 5 runs, but the 95th percentile data often tells the opposite story. Their method provides a "Significant Outperformance" metric that filters out this noise.
  • Simulation Costs: The study highlights a practical bottleneck: while quantum models are parameter-efficient, simulating them is computationally expensive. Training the 14-qubit model for 100 epochs took roughly 12 minutes per run, compared to mere seconds for classical models.

Limitations and Future Work

The biggest hurdle remains hardware. This study was conducted using noise-free simulations because current quantum computers are too "noisy" to run 100-run benchmarks reliably. While the authors proved that quantum models can be more efficient in theory, the real-world "runtime" advantage is still years away.

Future work involves testing these benchmarks on actual quantum hardware (NISQ devices) and exploring whether the parameter efficiency observed here holds up as the 6G environment scales to even more complex, multi-user scenarios. The authors also suggest that "quantum-inspired" classical algorithms might be a fruitful middle ground.

Summary updated

BibTeX
@inproceedings{meyer2025benchmarking,
  author = {Nico Meyer and Christian Ufrecht and George Yammine and Georgios D. Kontes and Christopher Mutschler and Daniel D. Scherer},
  title = {{Benchmarking Quantum Reinforcement Learning}},
  booktitle = {International Conference on Machine Learning},
  year = {2025},
  url = {https://openreview.net/forum?id=Ezp2elh9Yk},
}
Figure