Abstract
As part of a larger effort, this work-in-progress reports the possible advantages of modifying conventional workflows used to generate labelled training samples and train machine learning (ML) models on them. We compare results from three different workflows using neutron scattering data analysis as the motivating application and report about 20% improvement in speedup, with no appreciable loss of model accuracy, over a baseline workflow.
Introduction
In the field of materials science, neutron scattering serves as a critical diagnostic tool for uncovering the crystal structure of various materials. Researchers analyze "Bragg profiles"—two-dimensional patterns of scattered neutrons—to determine fundamental unit cell parameters such as lengths and angles. While Machine Learning (ML) models have demonstrated the ability to predict these structural parameters directly from scattering data, the development of such models is frequently hindered by a significant "data bottleneck."
Experimental labeled data is often scarce or difficult to obtain in sufficient quantities. Consequently, researchers rely on high-fidelity physics-based simulators, such as GSAS-II, to generate synthetic training datasets. However, these simulations are computationally intensive and time-consuming, often requiring hours or days to produce a dataset of sufficient scale. The traditional "baseline" approach to this problem is strictly sequential: the entire dataset is simulated first, and only once the data generation is complete does the training phase begin. This linear dependency often leads to the underutilization of expensive high-performance computing (HPC) resources.
Workflow Methodologies
To address these inefficiencies, this study evaluates three distinct workflow architectures implemented using the RADICAL-Ensemble Toolkit (EnTK). These workflows represent different strategies for managing the interaction between data simulation and model training.
- Baseline Workflow ($W_b$): This is the conventional sequential approach where all $N$ required samples are simulated in a single batch, followed by a monolithic training phase.
- Serial Phased Workflow ($W_s$): In this approach, data generation is divided into smaller "phases." The model is trained on an initial chunk of data; subsequently, a new chunk is simulated and appended to the training set for the next round of training. While still sequential, this phased approach improves memory management and provides a framework for future "active learning" implementations.
- Parallel Pipelined Workflow ($W_p$): This core innovation pipelines the phases to enable concurrency. While the ML model is training on data generated in Phase $k$, the simulator concurrently generates data for Phase $k+1$. This overlapping of tasks aims to minimize idle time on compute nodes.
Model Architecture and Data
The motivating application for these workflows is the analysis of Barium Titanate ($BaTiO_3$), a perovskite material that exists in three symmetry groups: tetragonal, trigonal, and cubic. The ML task involves classifying the symmetry group and regressing the unit cell parameters from the input Bragg profile.
The model architecture consists of a feature learner followed by fully connected layers. A Softmax layer handles the classification of the crystal system, while a parallel branch predicts the specific structural parameters. For this study, the largest experimental setup (E3) utilized 160,600 total samples. All experiments were conducted on the Theta supercomputer at Argonne National Laboratory, utilizing Intel KNL nodes.
Technical Innovation: Bias-Free Sampling
A significant challenge in incremental or phased training is "distribution shift." If a parameter range is simply partitioned into chunks, the model may become biased toward the most recent data or "forget" the distributions of previous phases. To mitigate this, the authors implemented a "bias-free" sampling policy. This strategy ensures that the data distribution in each phase remains representative of the entire parameter space. By carefully selecting non-overlapping samples that still span the global distribution, the researchers ensured that the model trained in the parallel workflow performed identically to the baseline model trained on the full dataset at once.
Key Results and Discussion
The experimental results demonstrate that the parallel pipelined workflow offers substantial performance gains without compromising scientific accuracy.
- Performance Speedup: The parallel workflow achieved a speedup ranging from 1.18x to 1.36x (roughly an 18% to 36% improvement) over the baseline across various experimental configurations.
- Accuracy Preservation: There was no appreciable loss in model accuracy. Test loss curves for the parallel and serial workflows converged asymptotically to the same levels as the baseline, validating the effectiveness of the bias-free sampling policy.
- Efficiency Gains: The logical separation of simulation and training tasks effectively emulates a heterogeneous environment. Even on CPU-only nodes, reducing the idle time between simulation and training phases leads to significant efficiency improvements.
Conclusion and Future Directions
This work demonstrates that optimizing the logic of ML workflows can yield significant performance improvements in materials science applications. However, the researchers noted that the current speedup is limited by a task imbalance: training currently takes approximately three times longer than simulation.
Future work will focus on integrating active learning policies, where the results of training in Phase $k$ dynamically inform the simulator on which parameter regions require more data in Phase $k+1$. Additionally, the implementation of early stopping and the utilization of dedicated GPU resources for training tasks are expected to further enhance the efficiency and scalability of the parallel pipelined workflow.
Summary updated