Low latency optical-based mode tracking with machine learning deployed on FPGAs on a tokamak

Yumou Wei, Ryan F. Forelli, Chris Hansen, Jeffrey P. Levesque, Nhan Tran, Joshua C. Agar, Giuseppe Di Guglielmo, Michael E. Mauel, Gerald A. NavratilarXiv 2023 · DOI · OpenReview

TL;DR

This paper presents a novel system for real-time tracking of plasma instabilities in tokamaks using high-speed camera data processed directly on an FPGA. By deploying a machine learning model on the camera's onboard FPGA, the system achieves microsecond-level latency, enabling immediate feedback for plasma control and opening doors for advanced diagnostics in fusion research and beyond.

What Problem Does This Paper Address?

Magnetic confinement fusion devices, like tokamaks, aim to harness the power of plasma. However, plasma can become unstable, threatening operation. To maintain stable conditions and achieve fusion-relevant parameters, active feedback control is crucial. This control requires rapid, non-invasive diagnostics that can detect instabilities and generate corrective signals within microseconds – a timescale dictated by the rapid evolution of plasma phenomena. Traditional methods often struggle to meet these stringent latency requirements, especially when dealing with the high data rates from modern optical diagnostics like high-speed cameras. The challenge lies in processing vast amounts of image data in real-time to extract critical information about plasma behavior and translate it into control actions with minimal delay.

Methods

The core innovation lies in deploying a machine learning model directly onto the Field-Programmable Gate Array (FPGA) of a high-speed camera's frame grabber. Specifically, the authors utilize a Convolutional Neural Network (CNN) trained to predict the amplitude and phase of the n=1 Magnetohydrodynamic (MHD) mode from camera images.

Key methodological aspects include:

  • Hardware Acceleration: Instead of offloading processing to a separate computer, the CNN inference is performed in situ on the FPGA. This drastically reduces data transfer bottlenecks and associated latencies.
  • High-Level Synthesis (HLS): The authors leverage the hls4ml toolchain to translate their TensorFlow CNN model into hardware description language (HDL) suitable for FPGA implementation. This bridges the gap between machine learning frameworks and hardware design.
  • Model Optimization for Hardware: Significant effort was dedicated to optimizing the CNN for the resource-constrained FPGA. This involved:
  • Quantization: Using arbitrary-precision fixed-point representations instead of full 32-bit floating-point numbers to reduce memory and computational requirements.
  • Sparsity: Applying pruning techniques to set a significant percentage of model weights to zero, further reducing computational load.
  • Reuse Factor Tuning: Adjusting the "reuse factor" for different layers to control the degree of parallelization and find a balance between latency and FPGA resource utilization.
  • Data Flow Management: Custom firmware was developed to handle image stream reordering (as camera data can arrive in non-sequential "stripes") and to embed model predictions back into the image stream for validation.

Datasets

The CNN models were trained and evaluated using a dataset collected from previous experiments on the High Beta Tokamak-Extended Pulse (HBT-EP) device. This dataset comprised:

  • Image Data: High-speed camera frames (250 kfps, 128x64 pixels, 12-bit grayscale) from 45 plasma discharges.
  • Ground Truth Data: Sine and cosine components of the n=1 MHD mode, derived from in-vessel low-field-side magnetic sensor arrays (FB3) using Fourier decomposition.

The dataset was split into training (40 shots) and testing (5 shots) sets, with further random splitting of the training set for validation.

Key Insights and Ideas

  • FPGA as a Real-time ML Inference Engine: The paper demonstrates that FPGAs, when coupled with appropriate HLS tools and optimization techniques, are highly capable of performing complex ML inference for real-time control applications, achieving latencies far beyond what is typically possible with CPUs or even GPUs for this specific task.
  • Co-design is Crucial: The success hinges on a tight co-design loop between the ML model architecture, its hardware implementation, and the specific constraints of the target FPGA. Simply porting a model is insufficient; significant optimization is required.
  • Optical Diagnostics for Control: Moving beyond traditional magnetic sensors, using high-speed optical camera data for real-time feedback control offers a non-invasive and flexible diagnostic approach, potentially simplifying system integration and maintenance.
  • Microsecond-Level Latency is Achievable: The demonstrated 17.6 µs trigger-to-output latency is a significant achievement, directly addressing the need for control signals that can keep pace with fast plasma dynamics.

Key Results

  • Low Latency: The system achieved a total trigger-to-output latency of 17.6 µs. This breaks down into:
  • 3 µs for camera exposure
    • 5.3 µs for camera readout and buffering
    • 7.7 µs for CNN model inference
    • 1.6 µs for serial writeout
  • High Throughput: The system can operate at a throughput of up to 120 kfps, with a minimum sampling interval of 8.3 µs.
  • Model Performance: The "Optimized" CNN model, while slightly reducing prediction accuracy compared to larger models, still significantly outperformed traditional non-deep-learning methods and met the stringent latency and resource requirements.
  • Resource Efficiency: The optimized model achieved a significant reduction in FPGA resource usage (e.g., ~36.7% LUTs, ~10.9% Registers, ~7.8% DSPs, ~9.3% BRAMs), making it feasible for deployment on the constrained frame grabber FPGA.

Limitations and Future Work

  • Single Mode Tracking: The current implementation is designed for tracking a single n=1 MHD mode. Future work will explore multi-mode control tasks, which would require more complex firmware and potentially alternative hardware configurations.
  • Fixed Control Parameters: The gain and phase shift parameters for the control signals are currently fixed. Future iterations will need the ability to adjust these parameters dynamically for different control campaigns.
  • Dataset Staleness: Recent upgrades to the HBT-EP device have altered plasma conditions, making the existing training dataset potentially obsolete. New data collection and model retraining will be necessary for future experiments.
  • Closed-Loop Control: While the system is designed for closed-loop control, actual feedback experiments have not yet been conducted. This requires integrating additional DAC and breakout board systems.
  • Domain Adaptation and Reinforcement Learning: The authors suggest exploring more advanced ML techniques like domain adaptation, active learning, and reinforcement learning for more robust and adaptive control systems, moving beyond supervised learning.
  • Broader Applications: The demonstrated workflow and hardware platform are applicable to a wide range of scientific domains beyond fusion, including cell classification, electron diffraction, and object detection.

Summary updated

Abstract

Active feedback control in magnetic confinement fusion devices is desirable to mitigate plasma instabilities and enable robust operation. Optical high-speed cameras provide a powerful, non-invasive diagnostic and can be suitable for these applications. In this study, we process fast camera data, at rates exceeding 100kfps, on $\textit{in situ}$ Field Programmable Gate Array (FPGA) hardware to track magnetohydrodynamic (MHD) mode evolution and generate control signals in real-time. Our system utilizes a convolutional neural network (CNN) model which predicts the $n$=1 MHD mode amplitude and phase using camera images with better accuracy than other tested non-deep-learning-based methods. By implementing this model directly within the standard FPGA readout hardware of the high-speed camera diagnostic, our mode tracking system achieves a total trigger-to-output latency of 17.6$\mu$s and a throughput of up to 120kfps. This study at the High Beta Tokamak-Extended Pulse (HBT-EP) experiment demonstrates an FPGA-based high-speed camera data acquisition and processing system, enabling application in real-time machine-learning-based tokamak diagnostic and control as well as potential applications in other scientific domains.
BibTeX
@misc{wei2023low,
  author = {Yumou Wei and Ryan F. Forelli and Chris Hansen and Jeffrey P. Levesque and Nhan Tran and Joshua C. Agar and Giuseppe Di Guglielmo and Michael E. Mauel and Gerald A. Navratil},
  title = {{Low latency optical-based mode tracking with machine learning deployed on FPGAs on a tokamak}},
  year = {2023},
  eprint = {2312.00128},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2312.00128},
}
Figure