Quantum reinforcement learning in continuous action space

Shaojun Wu, Shan Jin, Dingding Wen, D. J. Han, Xiaoting Wang

TL;DR

This paper introduces a novel quantum Deep Deterministic Policy Gradient (DDPG) algorithm designed to tackle reinforcement learning problems in continuous action spaces, a significant challenge for current quantum approaches. The method enables efficient, single-shot quantum state generation, meaning a single optimization process can produce a model capable of driving a system from a fixed initial state to any desired target state, a notable improvement over traditional methods.

What Problem Does This Paper Address?

Reinforcement learning (RL) is a powerful paradigm for sequential decision-making, but applying it to problems with continuous action spaces (CAS) is computationally challenging, especially on quantum computers. Existing quantum RL (QRL) methods have primarily focused on discrete action spaces. Extending them to CAS often involves discretizing the action space, which leads to the "curse of dimensionality" – an exponential increase in complexity with the number of actions. This paper aims to overcome this limitation by developing a QRL algorithm that can directly handle continuous actions, making QRL more applicable to complex quantum control and state generation tasks.

Methods

The core of this work is a Quantum Deep Deterministic Policy Gradient (Q-DDPG) algorithm. This approach adapts the classical DDPG algorithm for quantum computation. Key components include:

  • Quantum Neural Networks (QNNs): Used to represent the policy (mapping states to actions) and the value function (estimating the expected cumulative reward). These are implemented using variational quantum circuits (VQCs).
  • Continuous Action Space Handling: Unlike methods that discretize actions, Q-DDPG directly uses continuous action parameters. The agent's action is represented by a parameterized unitary operation.
  • Quantum State Generation Framework: The algorithm is framed as a state generation problem. The goal is to find a sequence of unitary operations that transforms a fixed initial quantum state into a desired target state.
  • Single-Shot Optimization: A crucial innovation is that after a one-time optimization of the QNNs, the resulting model can generate the required control sequence for any target state. This contrasts with conventional methods that require separate optimization for each target.
  • Experience Replay and Target Networks: Standard DDPG techniques are employed to stabilize training, including storing past experiences in a replay buffer and using target networks for more stable value estimation.

Datasets

Not discussed in the paper. The paper focuses on simulations of quantum systems rather than training on specific datasets.

Key Insights and Ideas

  • Bridging the Gap to Continuous Actions: The paper demonstrates that DDPG's actor-critic architecture can be effectively translated into a quantum framework, enabling QRL in continuous action spaces without the need for discretization.
  • "Single-Shot" Quantum State Generation: The most significant insight is the ability to perform state generation with a single optimization. This means a trained QRL agent can act as a universal controller for preparing various target states, drastically reducing the overhead compared to traditional methods that require re-optimization for each new target.
  • Potential for Quantum Control: The Q-DDPG framework offers a powerful new tool for designing quantum control sequences, which are essential for tasks like quantum error correction, quantum computing, and quantum simulation.
  • Addressing Near-Term Devices: The use of VQCs makes the proposed method compatible with current and near-term quantum hardware.

Key Results

  • Successful State Generation: Simulations show the Q-DDPG algorithm can efficiently generate target quantum states for one-qubit and two-qubit systems.
  • For a one-qubit case, the overlap with the target state reached $p_{50} \ge 0.99$ with a variance $\Delta(p_{50}) \le 4.47 \times 10^{-5}$ after 50 iterations.
  • For a two-qubit case (maximum entangled state), $p_{50} \ge 0.98$ with $\Delta(p_{50}) \le 4.04 \times 10^{-7}$.
  • Eigenvalue Problem Solved: The framework was also applied to solve the eigenvalue problem for Hamiltonians, demonstrating its versatility. For a one-qubit Hamiltonian, $p_{50} \ge 0.99$ with $\Delta(p_{50}) \le 1.98 \times 10^{-7}$.
  • Complexity Analysis: The paper provides an analysis of the quantum circuit and measurement complexity, suggesting that the number of measurements scales favorably with precision requirements.

Limitations and Future Work

  • Scalability: While simulations were performed for up to two qubits, the paper acknowledges that training QNNs becomes more challenging as the number of qubits increases. The use of a more advanced QNN architecture (from Ref. [62]) was necessary for the two-qubit case, hinting at the need for more sophisticated QNN designs for larger systems.
  • Noise: The simulations incorporated random noise in the reward calculation, but a comprehensive analysis of noise resilience and error mitigation for NISQ devices is not detailed.
  • Generalization to More Complex Systems: While the framework is promising, its performance on larger, more complex quantum systems with intricate dynamics remains an open question.
  • Future Work: The authors suggest exploring the application of their QRL method to other quantum control problems and investigating its potential for achieving exponential speedups over classical algorithms. They also highlight the open question of whether QRL can guarantee exponential speedups in terms of gate complexity.

Summary updated

Abstract

Quantum reinforcement learning (QRL) is a promising paradigm for near-term quantum devices. While existing QRL methods have shown success in discrete action spaces, extending these techniques to continuous domains is challenging due to the curse of dimensionality introduced by discretization. To overcome this limitation, we introduce a quantum Deep Deterministic Policy Gradient (DDPG) algorithm that efficiently addresses both classical and quantum sequential decision problems in continuous action spaces. Moreover, our approach facilitates single-shot quantum state generation: a one-time optimization produces a model that outputs the control sequence required to drive a fixed initial state to any desired target state. In contrast, conventional quantum control methods demand separate optimization for each target state. We demonstrate the effectiveness of our method through simulations and discuss its potential applications in quantum control.
BibTeX
@inproceedings{wu2025quantum,
  author = {Shaojun Wu and Shan Jin and Dingding Wen and D. J. Han and Xiaoting Wang},
  title = {{Quantum reinforcement learning in continuous action space}},
  booktitle = {Quantum},
  year = {2025},
  doi = {10.22331/q-2025-03-12-1660},
}
Figure