Quantum Multi-Agent Meta Reinforcement Learning

Won Joon Yun, Jihong Park, Joongheon Kim

Introduction

Multi-Agent Reinforcement Learning (MARL) presents a significant challenge in modern artificial intelligence, primarily due to the non-stationary nature of environments where multiple agents interact and evolve simultaneously. While meta-learning—the paradigm of "learning how to learn"—has emerged as a robust solution for rapid adaptation in classical domains, its integration into Quantum Machine Learning (QML) is still in its infancy. Current Quantum Neural Networks (QNNs) are constrained by the limitations of the Noisy Intermediate-Scale Quantum (NISQ) era, such as low qubit counts and high noise levels. This paper introduces QM2ARL, a framework designed to exploit the inherent architectural properties of QNNs to achieve efficient meta-learning and mitigate catastrophic forgetting in multi-agent systems.

Methodology: The Dyadic Advantage

The core innovation of QM2ARL lies in leveraging the "dyadic" structure of QNNs. Unlike classical neural networks, where all weights are mathematically homogeneous, QNN outputs are determined by two distinct sets of parameters: the circuit rotation angles ($\phi$) and the measurement basis or "poles" ($\theta$).

Overview of the QM2ARL training process, illustrating the transition from global meta-training to local pole adaptation and storage in pole memory.

The learning process is bifurcated into two stages:

  1. Meta-QNN Angle Training ($\phi$): Global knowledge and feature extraction logic are encoded into the Parameterized Quantum Circuit (PQC). This phase represents the computationally intensive "meta-training."
  2. Local QNN Pole Training ($\theta$): Adaptation to specific environmental variations is achieved solely by tuning the measurement poles on the Bloch sphere. Since a pole is defined by only two coordinates (polar and azimuthal angles), this allows for extremely fast, few-shot adaptation with minimal parameter updates.

Architecture: Decoupling Logic from Observation

The QM2ARL architecture consists of three primary stages: a state encoder, a PQC, and a measurement layer. The state encoder maps classical observations into quantum states, while the PQC applies a series of gates parameterized by $\phi$ to perform quantum logic. The final measurement layer projects the resulting state onto a basis defined by $\theta$.

The dyadic QNN architecture, highlighting the separation between the Parameterized Quantum Circuit (angles $\phi$) and the Measurement stage (poles $\theta$).

By treating the measurement as a tunable kernel, the agent can drastically alter its behavioral output without modifying the underlying quantum logic. This separation allows the meta-model to remain stable while the "decision head" (the pole) adapts to local environmental shifts.

Key Innovations

Angle-to-Pole Regularization

To ensure the meta-model generalizes across diverse tasks, the authors implement Angle-to-Pole Regularization. During the meta-training of $\phi$, artificial noise is injected into the pole domain ($\theta$). This forces the PQC to learn robust features that are not overly dependent on a specific measurement orientation. The paper provides theoretical proof that convergence is maintained despite this stochasticity.

Pole Memory

One of the most significant advantages of the dyadic approach is the efficiency of memory storage. To prevent catastrophic forgetting, agents can save their environment-specific pole parameters as low-dimensional "addresses." In QM2ARL, an agent can "reload" its expertise for a previously encountered environment almost instantly by resetting its measurement basis to the stored $\theta^*$ values, requiring only two parameters per agent per environment.

Experimental Evaluation and Results

The framework was rigorously tested in two environments: a cooperative Two-Step Game and a Single-Hop Offloading scenario for mobile edge computing.

Key findings include:

  • Rapid Adaptation: QM2ARL demonstrated significantly faster convergence in new environments compared to standard Quantum MARL baselines.
  • Performance Gains: In complex offloading tasks, the model achieved higher cumulative rewards than both classical MARL and standard QMARL models of equivalent parameter size.
  • Zero-Shot Recovery: Utilizing Pole Memory, agents could return to a previous environment and immediately regain peak performance, effectively eliminating the forgetting curve observed in traditional models.
  • Generalization Trade-off: While the regularization phase slightly increases initial training time, it yields a more versatile meta-model that stabilizes subsequent local adaptation.

Conclusion

QM2ARL represents a novel intersection of quantum mechanics and meta-learning. By decoupling the logic of the quantum circuit from the orientation of the measurement observer, the framework provides a lightweight, memory-efficient path for multi-agent systems to adapt to non-stationary environments. Future work will focus on scaling these dyadic properties to larger qubit systems and exploring more complex, continuous-action spaces.

Summary updated

Abstract

Although quantum supremacy is yet to come, there has recently been an increasing interest in identifying the potential of quantum machine learning (QML) in the looming era of practical quantum computing. Motivated by this, in this article we re-design multi-agent reinforcement learning (MARL) based on the unique characteristics of quantum neural networks (QNNs) having two separate dimensions of trainable parameters: angle parameters affecting the output qubit states, and pole parameters associated with the output measurement basis. Exploiting this dyadic trainability as meta-learning capability, we propose quantum meta MARL (QM2ARL) that first applies angle training for meta-QNN learning, followed by pole training for few-shot or local-QNN training. To avoid overfitting, we develop an angle-to-pole regularization technique injecting noise into the pole domain during angle training. Furthermore, by exploiting the pole as the memory address of each trained QNN, we introduce the concept of pole memory allowing one to save and load trained QNNs using only two-parameter pole values. We theoretically prove the convergence of angle training under the angle-to-pole regularization, and by simulation corroborate the effectiveness of QM2ARL in achieving high reward and fast convergence, as well as of the pole memory in fast adaptation to a time-varying environment.
BibTeX
@inproceedings{yun2023quantum,
  author = {Won Joon Yun and Jihong Park and Joongheon Kim},
  title = {{Quantum Multi-Agent Meta Reinforcement Learning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i9.26313},
}
Figure