Abstract: We implement a quantum error correction algorithm for bit-flip errors on the topological toric code using deep reinforcement learning. An action-value Q-function encodes the discounted value of moving a defect to a neighboring site on the square grid (the action) depending on the full set of defects on the torus (the syndrome or state). The Q-function is represented by a deep convolutional neural network. Using the translational invariance on the torus allows for viewing each defect from a central perspective which significantly simplifies the state space representation independently of the number of defect pairs. The training is done using experience replay, where data from the algorithm being played out is stored and used for mini-batch upgrade of the Q-network. We find performance which is close to, and for small error rates asymptotically equivalent to, that achieved by the Minimum Weight Perfect Matching algorithm for code distances up to $d=7$. Our results show that it is possible for a self-trained agent without supervision or support algorithms to find a decoding scheme that performs on par with hand-made algorithms, opening up for future machine engineered decoders for more general error models and error correcting codes.
Introduction: The Challenge of Quantum Error Correction
Quantum computers are notoriously fragile. Qubits are prone to "bit-flip" and "phase-flip" errors caused by environmental noise. To build a functional quantum computer, we use Quantum Error Correction (QEC).
The toric code is a popular topological QEC scheme where information is stored across a 2D lattice. When errors occur, they create "defects" (syndromes) at specific locations.
The challenge is the decoding problem: looking at these defects and figuring out exactly which physical qubits flipped so we can undo the damage. If the decoder makes the wrong guess and connects defects across the entire lattice, it causes a "logical failure," destroying the quantum information. While standard algorithms like Minimum Weight Perfect Matching (MWPM) work well for simple noise, they are "hand-made" and may struggle with more complex, realistic noise models. This research explores whether an AI can learn to be a decoder on its own.
Methods: Deep Q-Learning with a Central Perspective
The researchers implemented a Deep Q-Learning (DQN) approach where an agent learns to navigate the syndrome space.
- The Agent's Task: The agent observes the "syndrome"—the map of defects—and chooses an "action," which involves moving a defect one step on the grid to eventually meet and annihilate another defect.
- The Reward System: To encourage efficiency, the agent receives a reward of -1 for every step taken. This forces the agent to find the shortest possible path to clear defects, mirroring the goal of finding the most likely error chain.
- The "Perspective" Innovation: To prevent the state space from exploding as the lattice grows, the authors utilized the translational invariance of the torus. They created a "central perspective" for each defect, centering the view on that specific defect. This allows a single neural network to evaluate any defect regardless of its absolute position or the total number of defects.
The architecture uses a Deep Convolutional Neural Network (CNN) to process these perspectives, outputting "Q-values" that represent the expected future reward for moving in each cardinal direction.
Results: Performance and Thresholds
The agent was trained on synthetic datasets of bit-flip errors on toric codes with distances $d=3, 5,$ and $7$. Training occurred at a fixed 10% error rate, with testing performed across a range of noise levels up to 15%.
Key findings include:
- Near-Optimal Performance: For code distances up to $d=7$, the RL agent's success rate was nearly indistinguishable from the gold-standard MWPM algorithm.
- Threshold Behavior: The agent successfully captured the "threshold" behavior characteristic of the toric code. As the lattice size ($d$) increases, error correction becomes significantly more effective, provided the physical error rate remains below approximately 11%.
- Emergent Strategy: Without explicit instruction on MWPM rules, the agent naturally converged to a similar strategy, learning that the shortest path is typically the "correct" path for low error rates.
Discussion and Future Outlook
While the results for $d=7$ are promising, scaling to larger codes like $d=9$ presented challenges, likely requiring larger networks or more advanced training techniques. Furthermore, this study focused on simplified bit-flip errors and assumed perfect syndrome measurements.
The true potential of this reinforcement learning approach lies in its adaptability. Future work will focus on more complex, realistic noise models—such as depolarizing noise and noisy measurements—where traditional "hand-made" algorithms may falter. By learning directly from the environment, RL agents could provide the robust, machine-engineered decoders necessary for the next generation of quantum hardware.
Summary updated