TL;DR
This paper introduces a novel approach to controlling the Fermilab Booster accelerator using a digital twin and reinforcement learning (RL). By creating a highly accurate simulation of the accelerator's Gradient Magnet Power Supply (GMPS) system, they enable offline training of an RL agent, paving the way for more precise and robust control than traditional PID methods.
What Problem Does This Paper Address?
Controlling particle accelerators like the Fermilab Booster is a complex task. The Gradient Magnet Power Supply (GMPS) system, crucial for steering and accelerating the proton beam, is susceptible to variations caused by factors like electrical ground movement and ambient temperature changes. While a Proportional-Integral-Derivative (PID) controller currently compensates for these issues, it achieves only about 0.1% accuracy. The authors aim to significantly improve this precision by replacing the PID system with a reinforcement learning (RL) agent, which can learn optimal control strategies through interaction and feedback. However, training RL agents directly on a live accelerator is risky, necessitating a reliable offline training environment.
Methods
The core of this work lies in developing a robust "digital twin" of the GMPS control system. This digital twin is a machine learning model, specifically a stacked Long Short-Term Memory (LSTM) network, trained to accurately capture the complex, time-varying dynamics of the GMPS. LSTMs are well-suited for this task due to their ability to learn from sequential data and retain information over time.
Once the digital twin was validated, it was used as a simulated environment for training a Deep Q-Network (DQN) based RL agent. The agent's goal is to learn a policy that minimizes the difference between the observed and set minimum current of the GMPS. To manage the action space, the agent's control actions (adjustments to the GMPS voltage) were discretized into seven levels, including a zero-change option. The RL agent's policy is implemented as a multi-layer perceptron (MLP).
Datasets
The authors utilized approximately six months of time-series data from the Fermilab Booster complex, specifically selecting 250,000 continuous time steps from March 10, 2020. This data was used to train both the digital twin and the RL policy models. The paper also references a newly introduced dataset, "BOOSTR: A Dataset for Accelerator Control Systems" [12], which likely contains this data.
Key Insights and Ideas
- Digital Twin as a Safe RL Sandbox: The most critical insight is the successful creation of a high-fidelity digital twin that acts as a safe and reliable environment for training RL agents offline. This circumvents the significant risks associated with direct online RL training on a live accelerator.
- LSTM for Complex Dynamics: The choice of LSTMs proved effective in capturing the intricate, multi-frequency dynamics of the GMPS system, demonstrating their power in modeling complex physical processes.
- RL Outperforms PID: Preliminary results show a significant improvement in control performance when comparing the trained RL agent to the existing PID controller, suggesting a path towards much more precise accelerator operation.
- Importance of Data Validation: The paper emphasizes the rigorous validation process for the digital twin, highlighting that the accuracy of the simulation is paramount for the success of the subsequent RL training.
Key Results
The paper reports a factor of 2-4x improvement in control performance when comparing the trained DQN-GMPS system to the existing PID-GMPS controller. This improvement is visualized in Figure 2, which shows the total reward achieved by the RL agent during training and testing phases, indicating a successful learning process. The digital twin itself was validated through Mean-Squared Error (MSE) calculations, with the final chosen model achieving an MSE of 294.6336 (x10^-6).
Limitations and Future Work
The authors acknowledge that the current work is a prototype and that deploying the trained RL agent on the live system via an FPGA is the ultimate goal. A significant limitation discussed is the difficulty of implementing certain complex variable decomposition techniques (like Empirical Mode Decomposition) in real-time on the FPGA. Additionally, they had to develop their own MinMax scaler implementation to ensure compatibility with the FPGA. Future work will focus on the actual deployment of the RL agent onto the FPGA and further refining its performance. The paper also mentions the use of concrete dropout for uncertainty quantification, suggesting a direction for understanding the confidence of the RL agent's predictions.
Summary updated