Asynchronous training of quantum reinforcement learning

Samuel Yen-Chi Chen

Abstract: The development of quantum machine learning (QML) has received a lot of interest recently thanks to developments in both quantum computing (QC) and machine learning (ML). One of the ML paradigms that can be utilized to address challenging sequential decision-making issues is reinforcement learning (RL). It has been demonstrated that classical RL can successfully complete many difficult tasks. A leading method of building quantum RL agents relies on the variational quantum circuits (VQC). However, training QRL algorithms with VQCs requires significant amount of computational resources. This issue hurdles the exploration of various QRL applications. In this paper, we approach this challenge through asynchronous training QRL agents. Specifically, we choose the asynchronous training of advantage actor-critic variational quantum policies. We demonstrate the results via numerical simulations that within the tasks considered, the asynchronous training of QRL agents can reach performance comparable to or superior than classical agents with similar model sizes and architectures.

Introduction

Quantum Machine Learning (QML) is often touted for its potential to outperform classical algorithms, but it faces a massive practical hurdle: training time. In the realm of Reinforcement Learning (RL), where agents learn through millions of trial-and-error interactions, the computational overhead of simulating or running variational quantum circuits (VQCs) can be a significant bottleneck. Most existing Quantum RL (QRL) research relies on single-threaded training, where one agent interacts with one environment. This sequential approach is not only slow but can also be unstable because consecutive experiences in RL are highly correlated.

To address these challenges, this research introduces the Quantum Asynchronous Advantage Actor-Critic (QA3C) framework. By adapting the popular classical A3C framework to the quantum domain, the study demonstrates that quantum agents can be trained faster and more effectively using parallel processing.

The QA3C Framework

The QA3C framework utilizes a hybrid architecture known as a "dressed" VQC. In this setup, classical neural network layers serve as pre- and post-processing units for the quantum circuit. These classical layers handle dimensionality reduction of the input data and fine-tuning of the output, while the core decision-making logic is executed within an 8-qubit VQC.

Hybrid architecture of a 'dressed' variational quantum circuit (VQC) where classical neural networks (NN) process inputs and outputs.

The primary innovation lies in the asynchronous parallelism. Instead of a single agent, the framework spawns multiple "worker" agents (80 in this study) running on different CPU cores. Each worker interacts with its own copy of the environment and calculates its own gradients independently. These gradients are then uploaded asynchronously to a global "master" model.

The Quantum Asynchronous Advantage Actor-Critic (QA3C) framework featuring a global parameter server and multiple parallel workers.

This approach offers two major benefits. First, it significantly reduces total training time. Second, because workers are at different stages of their environments, the data they send back is diverse. This "de-correlates" the training data, leading to much more stable learning than a single-threaded agent could achieve.

Experimental Evaluation

The framework was tested on three classic benchmarks from the OpenAI Gym and MiniGrid suites:

  • Acrobot: A two-link pendulum where the agent must swing the end above a certain height.
  • Cart-Pole: The classic problem of balancing a pole on a moving cart.
  • MiniGrid-SimpleCrossing: A complex grid-world environment where the agent must navigate through gaps in walls to reach a goal. This environment is particularly challenging due to sparse rewards.

Key Results and Insights

Numerical simulations revealed that the QA3C approach provides impressive gains in both stability and efficiency:

  1. Stability through Diversity: By having 80 agents exploring different parts of the state space simultaneously, the global model avoids getting stuck in local optima that often plague single-threaded models.
  2. Quantum Parameter Efficiency: The quantum models used only 96 quantum parameters. Despite this small size, they matched or outperformed classical models with significantly more parameters (up to ~2,500 in the MiniGrid case). This suggests that quantum circuits may possess higher "expressive power" per parameter for specific decision-making tasks.
  3. Performance Benchmarks: In the Acrobot task, the quantum model achieved a stable, high score while the classical model struggled to converge. In the MiniGrid scenarios, the quantum agents demonstrated faster convergence and higher overall scores than classical benchmarks.

Conclusion

While the results are promising, several avenues for future work remain. The current study was performed via numerical simulations; running QA3C on actual Near-Term Intermediate-Scale Quantum (NISQ) hardware will introduce noise and latency issues that require further investigation. Additionally, as we scale to more qubits, the challenge of "barren plateaus" (vanishing gradients) must be addressed. The next logical step is the utilization of multiple physical Quantum Processing Units (QPUs) in parallel, moving QRL from simulation into high-performance quantum computing environments.

Summary updated

Abstract

The development of quantum machine learning (QML) has received a lot of interest recently thanks to developments in both quantum computing (QC) and machine learning (ML). One of the ML paradigms that can be utilized to address challenging sequential decision-making issues is reinforcement learning (RL). It has been demonstrated that classical RL can successfully complete many difficult tasks. A leading method of building quantum RL agents relies on the variational quantum circuits (VQC). However, training QRL algorithms with VQCs requires significant amount of computational resources. This issue hurdles the exploration of various QRL applications. In this paper, we approach this challenge through asynchronous training QRL agents. Specifically, we choose the asynchronous training of advantage actor-critic variational quantum policies. We demonstrate the results via numerical simulations that within the tasks considered, the asynchronous training of QRL agents can reach performance comparable to or superior than classical agents with similar model sizes and architectures.
BibTeX
@inproceedings{chen2023asynchronous,
  author = {Samuel Yen-Chi Chen},
  title = {{Asynchronous training of quantum reinforcement learning}},
  booktitle = {INNS DLIA@IJCNN},
  year = {2023},
  doi = {10.1016/j.procs.2023.08.171},
}
Figure