Reinforcement Learning in High-Energy Physics: A Survey from Accelerator Control to Theory Exploration

Luca AnzalonearXiv 2026 · arXiv

TL;DR

This paper provides the first comprehensive survey of how Reinforcement Learning (RL) is transforming High-Energy Physics (HEP), spanning the entire research pipeline from microsecond-level accelerator control to the abstract exploration of string theory. It identifies the specific "closed-loop" conditions where RL outperforms traditional algorithms, offering a strategic roadmap for AI-driven discovery in the next generation of particle colliders.

What Problem Does This Paper Address?

High-Energy Physics has long used Machine Learning, but mostly in the form of supervised learning—training models on fixed, labeled datasets to identify particles. However, many of the most critical tasks in physics are "closed-loop" decisions. For example, adjusting a magnet in a particle accelerator changes the beam's state, which in turn changes what the next adjustment should be.

Traditional methods like PID controllers or manual expert tuning struggle when these systems become high-dimensional, non-differentiable, or subject to "drift" (changes over time due to temperature or wear). This survey addresses the need for a unified framework to apply RL—a paradigm where agents learn through trial and error—to these complex, sequential decision-making problems across all HEP sub-fields.

Figure 1: RL decision points along the HEP workflow, each labeled with the decision taken by the agent and the section that reviews it. Run-time applications act on the accelerator and on the data stream, offline applications act on recorded data and on theory, and detector design precedes data taking. Dashed arrows denote indirect influence: the designed detector determines the data that reach the trigger, and theory predictions inform the analyses that constrain them.

Methods

The paper categorizes RL applications into three recurring formulations:

  1. Control Loops: Agents managing physical systems over time (e.g., beam steering, trigger thresholds).
  2. Human-in-the-Loop: Agents monitoring data streams and deciding when to alert a human expert (e.g., Data Quality Monitoring).
  3. Sequential Construction: Agents building a physics object "edit-by-edit" (e.g., reconstructing a jet or designing a detector layout).

The survey reviews a vast toolkit of algorithms, from standard Deep Q-Networks (DQN) for discrete tasks to Soft Actor-Critic (SAC) and PPO for continuous control. It highlights Model-Based RL, where an agent uses a "digital twin" (a simulator) to plan actions without risking real hardware, and Physics-Informed RL, where laws of physics are baked directly into the agent’s reward function.

Datasets

The reviewed research utilizes a mix of high-fidelity simulators and real-world experimental data:

  • Simulators: Standard physics tools like Geant4 (detector response), Elegant and Cheetah (accelerator optics), and Pythia (event generation).
  • Real-World Data: Policies deployed on live accelerators at CERN (AWAKE, LINAC4) and DESY (ARES), as well as recorded collision data from the CMS experiment at the Large Hadron Collider (LHC).
  • Benchmarks: The paper points to emerging open resources like the Ginkgo jet model and the MEDIC fault-injection framework for data quality monitoring.

Key Insights and Ideas

  • The "Aha" Moment of Amortization: While RL training is computationally expensive, the resulting "policy" is a simple neural network that can make decisions in microseconds. This makes it uniquely suited for the LHC’s Level-1 triggers, which must decide which data to keep every 25 nanoseconds.
  • Robustness over Precision: Unlike classical controllers that might fail if a single sensor breaks, RL agents trained over a distribution of "noisy" conditions tend to degrade gracefully, finding creative ways to stabilize a beam even with faulty hardware.
  • Physics as the Label: In many cases, we don't have "ground truth" labels. RL allows researchers to use physical consistency (like the likelihood of Coulomb scattering) as a reward signal, effectively letting the agent learn from the laws of nature rather than human-provided examples.

Key Results

  • Accelerator Control: At the ARES linac (DESY), RL-trained optimizers achieved lower final beam errors than Bayesian Optimization and were orders of magnitude faster at inference.
  • Triggering: In a milestone result, RL-based trigger control was demonstrated on real CMS collision data, significantly improving rate stability compared to traditional PID baselines.
  • Theory Exploration: In string theory model building, RL agents found consistent physical models up to 1,700x more efficiently than random searches.
  • Tracking: Multi-agent RL systems have successfully reconstructed particle tracks in high-density environments by treating each track as a competing agent.

Limitations and Future Work

The authors identify several "open hurdles" for the field:

  • The Sim-to-Real Gap: A policy that works perfectly in a simulator often fails on a real machine because no simulation is perfect. Techniques like "domain randomization" are helping, but it remains a primary challenge.
  • Safety Guarantees: Currently, most RL agents are kept "safe" through simple reward penalties. For high-stakes hardware, the field needs to move toward formal Constrained MDPs that provide mathematical guarantees against damaging equipment.
  • Interpretability: Physicists are naturally skeptical of "black box" neural networks. Future work must focus on Explainable RL (XRL)—distilling complex policies into readable decision trees or symbolic formulas that humans can verify.

Summary updated

Abstract

Reinforcement learning (RL) is a machine learning paradigm in which an agent learns a decision-making policy by interacting with an environment, guided by a scalar reward rather than explicit supervision. Supervised and unsupervised learning have become indispensable tools in high-energy physics (HEP), but they learn from fixed datasets, whereas many HEP tasks are closed-loop decisions: the data an algorithm observes depend on its choices, the feedback is sparse or delayed, and the objective is often measured on a real machine or in a stochastic simulation, not by a differentiable model. This survey reviews the application of RL to such tasks: accelerator control and feedback, data quality monitoring, jet grooming, clustering, and representation, charged particle tracking, beyond-the-Standard-Model (BSM) and string model building, detector design, trigger control, analysis, and formal quantum field theory computation. It provides a self-contained introduction to the underlying RL methods, from model-free and model-based to multi-agent, offline, inverse, and bandit approaches, reviews each domain against its traditional methods, and summarizes the head-to-head comparisons. The applications range from policies deployed on real accelerators and evaluated on recorded collision data to exploratory studies in formal theory. RL is most advantageous for repeated tasks with a fast simulator, decisions inside fast control loops, operation under faults and corrupted measurements, physics-based rewards in place of labels, and sequential discrete constructions with a reusable policy, while Bayesian optimization, genetic algorithms, and differentiable methods remain competitive elsewhere. We conclude with the open challenges that recur across domains, including reward design, sim-to-real transfer, safety, the lack of shared benchmarks, and the statistical validity of policy-driven decisions.
BibTeX
@misc{anzalone2026reinforcement,
  author = {Luca Anzalone},
  title = {{Reinforcement Learning in High-Energy Physics: A Survey from Accelerator Control to Theory Exploration}},
  year = {2026},
  eprint = {2611.02049},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.02049},
}
Figure