Dynamic Inhomogeneous Quantum Resource Scheduling with Reinforcement Learning

Linsen Li, Pratyush Anand, Kaiming He, Dirk R. Englund

Abstract

Building large-scale quantum computers requires connecting qubits into complex "resource states," a process hindered by hardware inhomogeneity and the probabilistic nature of quantum entanglement. We present a Reinforcement Learning (RL) framework utilizing a Transformer-based architecture, "Transformer-on-QuPairs," to manage these dynamic quantum resources in real-time. By processing relationships between qubit pairs and accounting for decoherence, the RL agent achieves a 3x performance improvement over traditional scheduling methods. Our results demonstrate that intelligent scheduling is essential for scaling quantum systems, with performance gains increasing as the number of qubits grows.

Introduction

The construction of large-scale quantum computers relies on entangling qubits into specific, high-fidelity patterns known as "cluster states." However, real-world hardware faces two significant challenges: inhomogeneity and probabilistic outcomes. Inhomogeneity arises from fabrication limits, where qubits vary in coherence time and connectivity. Probabilistic outcomes occur because entangling protocols, such as heralded remote entanglement, are stochastic; failures require retries while existing entanglement decays due to decoherence.

Evolution of a cluster state over time steps (a-c) and the corresponding scheduling table (d), illustrating the growth of the maximum cluster size despite probabilistic failures.

Deciding which qubits to pair dynamically is an NP-hard scheduling problem. Static rules fail to adapt to the "ticking clock" of decoherence, where a delay in finding a perfect link can lead to the loss of the entire quantum state. Traditional optimization methods like Static Minimum Spanning Trees (MST) struggle in these environments because they cannot account for the "luck of the draw" in entanglement successes.

Methods

The proposed solution treats scheduling as a game played in a digitized environment, modeled via Monte Carlo Simulations (MCS). The core of the system is the "Transformer-on-QuPairs" architecture, which leverages the self-attention mechanism to identify optimal entanglement paths in a non-uniform grid.

The RL framework (a) showing the agent-environment loop, the Transformer-on-QuPairs architecture (b) with multi-head attention, and a heatmap (c) of the predicted error for qubit pairs.

The model incorporates several key components:

  • Input Encoding: The agent receives "pre-information" matrices $M_F$ and $M_R$ representing hardware fidelity and success rates, alongside a dynamic state matrix $M_S$ representing the current entanglement network.
  • Self-Attention: The Transformer uses multi-head attention to weigh the importance of different qubit pairs, predicting the "cost" or expected error $\epsilon(t)$ of attempting a specific entanglement link.
  • Reward Function: The agent is trained to maximize the Quantum Volume ($V_Q$), a metric derived from the maximum cluster size $N_{max}(t)$ and the accumulated system error.
  • Dynamic Guidance: Unlike static maps, this RL agent provides real-time, next-step guidance, choosing the path that balances the quality of a link with the speed of construction to beat decoherence.

Results and Discussion

The RL agent was benchmarked against random scheduling, Static MST, and greedy heuristics using high-fidelity simulated datasets grounded in real-world physics (e.g., spin-photon interfaces).

  • Performance Gain: The Transformer-on-QuPairs scheduler improved system performance by over 300% compared to rule-based greedy agents.
  • Adaptability: In highly inhomogeneous environments, the RL approach significantly outperformed Static MST. While MST works well for perfect hardware, it lacks the flexibility to handle the stochastic failures inherent in real quantum systems.
  • Scalability and Transfer Learning: As the system size increased from 40 to 160 qubits, the performance gap between the RL agent and the random baseline grew from 3.20x to 4.56x. Furthermore, agents trained on smaller systems could be effectively "upscaled" to larger systems, suggesting that the learned scheduling strategies are generalizable.

Conclusion

This work establishes RL-driven scheduling as a critical component for the next generation of scalable quantum processors. While the $N^2$ scaling of qubit pairs may present computational bottlenecks for massive systems, future integration with Monte Carlo Tree Search (MCTS) could provide even better look-ahead capabilities. Intelligent, real-time management of "messy" qubits is a promising path toward practical, large-scale quantum computation.

Summary updated

Abstract

A central challenge in quantum information science and technology is achieving real-time estimation and feedforward control of quantum systems. This challenge is compounded by the inherent inhomogeneity of quantum resources, such as qubit properties and controls, and their intrinsically probabilistic nature. This leads to stochastic challenges in error detection and probabilistic outcomes in processes such as heralded remote entanglement. Given these complexities, optimizing the construction of quantum resource states is an NP-hard problem. In this paper, we address the quantum resource scheduling issue by formulating the problem and simulating it within a digitized environment, allowing the exploration and development of agent-based optimization strategies. We employ reinforcement learning agents within this probabilistic setting and introduce a new framework utilizing a Transformer model that emphasizes self-attention mechanisms for pairs of qubits. This approach facilitates dynamic scheduling by providing real-time, next-step guidance. Our method significantly improves the performance of quantum systems, achieving more than a 3$\times$ improvement over rule-based agents, and establishes an innovative framework that improves the joint design of physical and control systems for quantum applications in communication, networking, and computing.
BibTeX
@misc{li2024dynamic,
  author = {Linsen Li and Pratyush Anand and Kaiming He and Dirk R. Englund},
  title = {{Dynamic Inhomogeneous Quantum Resource Scheduling with Reinforcement Learning}},
  year = {2024},
  eprint = {2405.16380},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2405.16380},
}
Figure