Variational Quantum Circuits and Deep Reinforcement Learning

Samuel Yen-Chi Chen, Hsi-Sheng Goan

Abstract

The state-of-the-art machine learning approaches are based on classical von Neumann computing architectures and have been widely used in many industrial and academic domains. With the recent development of quantum computing, researchers and tech-giants have attempted new quantum circuits for machine learning tasks. However, the existing quantum computing platforms are hard to simulate classical deep learning models or problems because of the intractability of deep quantum circuits. Thus, it is necessary to design feasible quantum algorithms for quantum machine learning for noisy intermediate scale quantum (NISQ) devices. This work explores variational quantum circuits for deep reinforcement learning. Specifically, we reshape classical deep reinforcement learning algorithms like experience replay and target network into a representation of variational quantum circuits. Moreover, we use a quantum information encoding scheme to reduce the number of model parameters compared to classical neural networks. To the best of our knowledge, this work is the first proof-of-principle demonstration of variational quantum circuits to approximate the deep $Q$-value function for decision-making and policy-selection reinforcement learning with experience replay and target network. Besides, our variational quantum circuits can be deployed in many near-term NISQ machines.

Introduction

Deep Reinforcement Learning (DRL) has revolutionized fields from complex game-playing to robotic control. However, these successes rely on computationally intensive classical neural networks. As we enter the Noisy Intermediate-Scale Quantum (NISQ) era, researchers are looking for ways to harness quantum advantages. The primary challenge lies in the fact that current quantum hardware is prone to noise and limited in qubit count, making it impossible to run the "deep" quantum circuits that would directly mirror classical deep learning architectures.

This paper addresses this gap by introducing a quantum-native reinforcement learning algorithm designed to be "shallow" enough for today's hardware while remaining powerful enough for sequential decision-making.

Methods: The Variational Quantum Deep Q-Network (VQ-DQN)

The authors propose the Variational Quantum Deep Q-Network (VQ-DQN), a hybrid quantum-classical approach that replaces the classical neural network with a Variational Quantum Circuit (VQC) to approximate the $Q$-value function.

  1. Quantum Function Approximation: The VQC predicts the expected reward for actions in a given state.
  2. State Encoding: Classical data (such as grid coordinates) is converted into quantum states using computational basis encoding.
  3. Circuit Architecture: The model utilizes rotational gates ($R_x, R_z$) for state preparation and CNOT gates to create entanglement, allowing the model to represent complex relationships with significantly fewer parameters than classical counterparts.
  4. Classical Optimization: A classical computer runs the training loop using the RMSprop optimizer, updating the quantum circuit's parameters based on environmental rewards.
  5. DRL Stability: The framework successfully integrates Experience Replay and Target Networks, two critical techniques from classical DRL that prevent the learning process from diverging.

Experimental Environments

The agents were evaluated on two primary benchmarks to test navigation and communication capabilities:

  • Frozen-Lake (OpenAI Gym): A navigation task where an agent must cross a 4x4 grid of ice to reach a goal while avoiding hazardous holes.
  • Cognitive-Radio: A communication task where a "secondary user" (the agent) must learn to select open frequency channels to avoid collisions with a "primary user" following a periodic pattern.
Frozen-Lake environments (a), (b), and (c) used to evaluate the quantum agent's navigation capabilities.

Key Insights and Results

The Parameter Advantage

A striking finding of this research is the efficiency of the VQ-DQN. The number of trainable parameters scales linearly $O(n)$ with input size, whereas classical DQNs often scale quadratically or cubically. For a 4-channel radio task, the quantum model required only 28 parameters, compared to 64 for a standard tabular approach.

Architecture of the Quantum RL Agent using Variational Quantum Circuits for spectrum management in Cognitive-Radio tasks.

Resilience to Noise

Because the VQC is trained iteratively by a classical optimizer, the system can "learn" to compensate for hardware noise. Simulations using the noise profiles of IBM’s 20-qubit Poughkeepsie machine showed the model remained stable. Furthermore, the researchers successfully validated their models on the IBM Q Valencia hardware, proving that these quantum agents can operate on real-world NISQ devices today.

Performance and Convergence

In the Frozen-Lake task, the quantum agent reached an optimal policy within approximately 200 episodes. In the Cognitive-Radio task, the agent achieved near-perfect scores (100/100) across various channel configurations, often converging faster than classical methods.

Conclusion and Future Work

This work provides a successful proof-of-principle for using VQCs in reinforcement learning. While current experiments are limited to discrete state spaces due to the computational cost of simulating quantum circuits on classical hardware, the results are promising. Future research will explore amplitude encoding to handle exponentially larger datasets and aim to scale VQ-DQNs to continuous state spaces and high-dimensional inputs like raw video frames.

Summary updated

Abstract

The state-of-the-art machine learning approaches are based on classical von Neumann computing architectures and have been widely used in many industrial and academic domains. With the recent development of quantum computing, researchers and tech-giants have attempted new quantum circuits for machine learning tasks. However, the existing quantum computing platforms are hard to simulate classical deep learning models or problems because of the intractability of deep quantum circuits. Thus, it is necessary to design feasible quantum algorithms for quantum machine learning for noisy intermediate scale quantum (NISQ) devices. This work explores variational quantum circuits for deep reinforcement learning. Specifically, we reshape classical deep reinforcement learning algorithms like experience replay and target network into a representation of variational quantum circuits. Moreover, we use a quantum information encoding scheme to reduce the number of model parameters compared to classical neural networks. To the best of our knowledge, this work is the first proof-of-principle demonstration of variational quantum circuits to approximate the deep $Q$-value function for decision-making and policy-selection reinforcement learning with experience replay and target network. Besides, our variational quantum circuits can be deployed in many near-term NISQ machines.
BibTeX
@article{chen2020variational,
  author = {Samuel Yen-Chi Chen and Hsi-Sheng Goan},
  title = {{Variational Quantum Circuits and Deep Reinforcement Learning}},
  journal = {IEEE Access},
  year = {2020},
  doi = {10.1109/access.2020.3010470},
}
Figure