Asynchronous training of quantum reinforcement learning

Samuel Yen-Chi Chen

Quantum Machine Learning (QML) is often touted for its potential to outperform classical algorithms, but it faces a massive practical hurdle: training time. In the realm of Reinforcement Learning (RL), where agents learn through millions of trial-and-error interactions, the computational overhead of simulating or running variational quantum circuits (VQCs) can be a dealbreaker.

A recent paper by Samuel Yen-Chi Chen (Wells Fargo) titled "Asynchronous training of quantum reinforcement learning" proposes a solution. By adapting the popular classical "Asynchronous Advantage Actor-Critic" (A3C) framework to the quantum domain, the author demonstrates that we can train quantum agents faster and more effectively using parallel processing.

TL;DR

This paper introduces QA3C, a framework that trains multiple quantum reinforcement learning agents in parallel and asynchronously. By sharing gradients across many "workers," the method significantly reduces training time and improves learning stability, allowing small quantum models to outperform classical counterparts on standard benchmarks.

What Problem Does This Paper Address?

Reinforcement Learning agents learn by interacting with an environment to maximize a reward. Most current Quantum RL (QRL) research relies on Variational Quantum Circuits (VQCs)—trainable quantum "neural networks."

However, VQCs are notoriously slow to train. Most existing QRL studies use single-threaded training, where one agent interacts with one environment. This is a bottleneck: if the quantum simulation is slow, the agent takes ages to learn. Furthermore, on-policy RL (where the agent learns from its current experience) can be unstable because consecutive experiences are highly correlated.

The challenge is: How do we make QRL training efficient enough for complex, real-world tasks?

Methods

The author proposes Quantum Asynchronous Advantage Actor-Critic (QA3C). Here’s how it works:

  1. The "Dressed" VQC: The agent uses a hybrid architecture. Classical neural network layers "dress" the quantum circuit, handling dimensionality reduction of the input data and fine-tuning the output. The core decision-making, however, happens in an 8-qubit VQC.
  2. Asynchronous Parallelism: Instead of one agent, the framework spawns multiple "worker" agents (80 in this study) running on different CPU cores.
  3. Global Shared Memory: There is a "master" model. Each worker interacts with its own copy of the environment, calculates its own gradients (the direction to improve the model), and asynchronously uploads those gradients to the master model.
  4. The Advantage: Because workers are at different stages of their environments, the data they send back is diverse. This "de-correlates" the data, leading to much more stable training than a single agent could achieve.

Datasets

The framework was tested on three classic benchmarks from the OpenAI Gym and MiniGrid suites:

  • Acrobot: A two-link pendulum where the agent must swing the end above a certain height.
  • Cart-Pole: The classic "balance a pole on a cart" problem.
  • MiniGrid-SimpleCrossing: A more complex grid-world environment where the agent must find a path to a goal through gaps in walls. This environment is particularly tough because rewards are "sparse"—the agent gets nothing until it actually reaches the goal.

Key Insights and Ideas

  • Stability through Diversity: The most significant "aha moment" is that asynchronicity isn't just about speed; it's about robustness. By having 80 agents exploring different parts of the state space simultaneously, the global model avoids getting stuck in local optima that plague single-threaded classical models.
  • Quantum Parameter Efficiency: The quantum models used only 96 quantum parameters. Despite this tiny size, they matched or beat classical models with significantly more parameters (up to ~2,500 in the MiniGrid case). This suggests that quantum circuits may have a higher "expressive power" per parameter for certain decision-making tasks.
  • Hybrid is Key: The use of "dressing" layers (classical NNs before and after the VQC) allows the quantum circuit to focus on the high-level logic of the policy while the classical parts handle the "grunt work" of data formatting.

Key Results

The numerical simulations showed impressive gains for the QA3C approach:

  • Acrobot: The quantum model achieved a stable, high score while the classical model struggled to converge and showed significant performance collapses.
  • Cart-Pole: While the classical model learned faster in the very early stages, the quantum model eventually surpassed it, reaching a significantly higher peak score.
  • MiniGrid: In the most difficult "SimpleCrossing" scenarios (S9N2 and S9N3), the quantum agents demonstrated faster convergence and higher overall scores than the classical benchmarks.

Limitations and Future Work

While the results are promising, there are a few caveats:

  • Simulation vs. Hardware: The study was performed via numerical simulations on classical CPUs. Running this on actual Near-Term Intermediate-Scale Quantum (NISQ) hardware would introduce noise and latency issues not fully captured here.
  • Scaling Qubits: The study used 8 qubits. As we scale to more qubits, QML models often face "barren plateaus" (where gradients vanish, making learning impossible). Whether QA3C helps mitigate barren plateaus remains an open question.
  • Future Direction: The author suggests that the next logical step is utilizing multiple Quantum Processing Units (QPUs) in parallel, which would move QRL from the realm of simulation into high-performance quantum computing.

Summary updated

Abstract

The development of quantum machine learning (QML) has received a lot of interest recently thanks to developments in both quantum computing (QC) and machine learning (ML). One of the ML paradigms that can be utilized to address challenging sequential decision-making issues is reinforcement learning (RL). It has been demonstrated that classical RL can successfully complete many difficult tasks. A leading method of building quantum RL agents relies on the variational quantum circuits (VQC). However, training QRL algorithms with VQCs requires significant amount of computational resources. This issue hurdles the exploration of various QRL applications. In this paper, we approach this challenge through asynchronous training of QRL agents. Specifically, we choose the asynchronous training of advantage actor-critic variational quantum policies. We demonstrate the results via numerical simulations that within the tasks considered, the asynchronous training of QRL agents can reach performance comparable to or superior to classical agents with similar model sizes and architectures.
BibTeX
@inproceedings{chen2023asynchronous,
  author = {Samuel Yen-Chi Chen},
  title = {{Asynchronous training of quantum reinforcement learning}},
  booktitle = {INNS DLIA@IJCNN},
  year = {2023},
  doi = {10.1016/j.procs.2023.08.171},
}

Loading related content...

Figure