Integrated edge-to-exascale workflow for real-time steering in neutron scattering experiments

Junqi Yin, Viktor Reshniak, Siyan Liu, Guannan Zhang, Xiaoping Wang, Zhongcan Xiao, Zachary Morgan, Sylwia Pawlędzio, Thomas Proffen, Christina Hoffmann, Huibo Cao, Bryan C. Chakoumakos, Yaohua LiuStructural 2024 · DOI

TL;DR

This paper introduces an "edge-to-exascale" computational workflow that uses AI and supercomputing to steer neutron scattering experiments in real-time. By predicting scattering patterns with a Temporal Fusion Transformer model and processing data at the edge and on the Frontier supercomputer, they significantly speed up analysis and enable dynamic adjustments to experiments, optimizing beam time and improving accuracy.

What Problem Does This Paper Address?

Neutron scattering experiments are powerful tools for understanding material structures and dynamics. However, they generate massive amounts of high-dimensional "event-mode" data. Traditionally, analyzing this data and making adjustments to the experiment happens after data collection, which is inefficient and limits the ability to optimize experiments in real-time, especially for complex studies like crystallography. This delay means valuable beam time can be wasted, and crucial experimental parameters might be missed. The challenge is to process this data fast enough to provide actionable feedback during the experiment, allowing for dynamic steering and optimization.

Methods

The core of this work is an "edge-to-exascale" workflow.

  • Edge Computing: At the neutron source (SNS), data is preprocessed rapidly on local systems (like DGX workstations) using tools like ADARA for live streaming. This "edge" processing filters and reduces the data volume before it's sent further.
  • Exascale Computing: The heavy lifting of training sophisticated AI models happens on the Frontier supercomputer. This allows for the large-scale training required for complex models.
  • Temporal Fusion Transformer (TFT): This deep learning model is central to the workflow. It's designed for time-series forecasting and is used here to predict 4D neutron scattering patterns with voxel-level precision. It can handle diverse input features like neutron count rates and time-of-flight data.
  • Distributed Training: To leverage Frontier's power, the training process is distributed across its many GPUs.
  • Real-time Feedback Loop: The workflow is designed to continuously process incoming data, make predictions, evaluate errors, and then feed this information back to adjust experimental parameters in real-time.

Datasets

The paper uses measured event data from the Spallation Neutron Source (SNS) TOPAZ beamline. This data is in NeXus format, which is a standard for neutron scattering data. The paper doesn't introduce a new benchmark dataset but rather demonstrates the workflow on existing experimental data.

Key Insights and Ideas

  • Bridging the Gap: The most significant insight is the successful integration of edge and exascale computing to create a functional real-time steering workflow. This overcomes the traditional bottleneck of post-experiment analysis.
  • TFT for Prediction: The Temporal Fusion Transformer proves to be a powerful tool for predicting complex, multi-dimensional neutron scattering patterns, enabling proactive adjustments.
  • Efficiency Gains: By processing data at the edge and using exascale resources for training, the workflow drastically reduces data processing time from hours to minutes.
  • Adaptive Experimentation: The ability to dynamically steer experiments based on real-time predictions opens up new possibilities for exploring complex materials and optimizing data collection strategies.
  • Scalability: The workflow is designed with scalability in mind, aiming for broader adoption across different neutron scattering facilities.

Key Results

  • Reduced Processing Time: The workflow offers a substantial reduction in data processing time, from hours to minutes, through distributed training.
  • Improved Accuracy: Significant improvements in model accuracy are reported, with a relative error of just 2.8% in real-time predictions, well within the target of <5%.
  • Beam Time Savings: The approach has the potential to save up to 29% of neutron beam time by avoiding unnecessary data collection once predictions align with experimental results.
  • Efficient Data Handling: The edge-to-exascale approach optimizes memory usage and parallelization by processing data in smaller, manageable subsets across multiple ranks.

Limitations and Future Work

  • Proof-of-Concept Stage: The authors acknowledge that real-time experiment steering is still in the proof-of-concept stage.
  • Live Data Integration: While the workflow is demonstrated with stored data, future work will focus on integrating live data streams more seamlessly.
  • Full Automation: The ultimate goal is to achieve fully automated real-time steering.
  • Dynamic Feedback Loops: Future research will explore integrating dynamic feedback loops for responses to external stimuli like temperature, pressure, or magnetic fields.
  • Generalizability: The workflow's generalizability will be tested across multiple experiments and instruments.
  • Statistical Metrics: Further statistical metrics will be gathered to validate the workflow's performance.

Summary updated

Abstract

We introduce a computational framework that integrates artificial intelligence (AI), machine learning, and high-performance computing to enable real-time steering of neutron scattering experiments using an edge-to-exascale workflow. Focusing on time-of-flight neutron event data at the Spallation Neutron Source, our approach combines temporal processing of four-dimensional neutron event data with predictive modeling for multidimensional crystallography. At the core of this workflow is the Temporal Fusion Transformer model, which provides voxel-level precision in predicting 3D neutron scattering patterns. The system incorporates edge computing for rapid data preprocessing and exascale computing via the Frontier supercomputer for large-scale AI model training, enabling adaptive, data-driven decisions during experiments. This framework optimizes neutron beam time, improves experimental accuracy, and lays the foundation for automation in neutron scattering. Although real-time experiment steering is still in the proof-of-concept stage, the demonstrated potential of this system offers a substantial reduction in data processing time from hours to minutes via distributed training, and significant improvements in model accuracy, setting the stage for widespread adoption across neutron scattering facilities and more efficient exploration of complex material systems.
BibTeX
@inproceedings{yin2024integrated,
  author = {Junqi Yin and Viktor Reshniak and Siyan Liu and Guannan Zhang and Xiaoping Wang and Zhongcan Xiao and Zachary Morgan and Sylwia Pawlędzio and Thomas Proffen and Christina Hoffmann and Huibo Cao and Bryan C. Chakoumakos and Yaohua Liu},
  title = {{Integrated edge-to-exascale workflow for real-time steering in neutron scattering experiments}},
  booktitle = {Structural},
  year = {2024},
  doi = {10.1063/4.0000279},
}
Figure