Abstract
Particle track reconstruction in high-energy physics (HEP) is a computationally intensive task critical for interpreting data from experiments like the ATLAS detector at the Large Hadron Collider. While Graph Neural Networks (GNNs) have emerged as a scalable alternative to traditional algorithms, they are often limited by GPU memory when processing large collision events. This paper introduces a minibatch subgraph-sampling approach to the Exa.TrkX pipeline, utilizing the ShaDow algorithm to process large graphs without data loss. By incorporating matrix-based bulk sampling and coalesced all-reduce optimizations, the new pipeline achieves higher reconstruction accuracy and up to a 2x training speedup, demonstrating superlinear scaling on massive datasets.
Introduction
In high-energy physics experiments, detectors record millions of 3D coordinates, or "hit points," during subatomic particle collisions. Reconstructing the paths of these particles—a process known as track reconstruction—is essential for downstream analysis. However, traditional reconstruction algorithms scale superlinearly with the number of hits, creating a significant computational bottleneck as detector luminosity increases.
The Exa.TrkX project addresses this by framing track reconstruction as an edge classification problem on graphs. GNNs are employed to predict which hit points connect to form valid tracks. Despite their potential, current implementations often rely on "full-graph training," where the entire particle graph is loaded into GPU memory. Because GNNs require storing large activation matrices for backpropagation, massive collision events frequently exceed memory limits. To prevent system crashes, the standard pipeline simply skips these large graphs. This practice is doubly detrimental: it discards valuable training data and restricts the model to a large-batch training regime, which often generalizes more poorly than small-batch stochastic training.
Methodology
To overcome memory constraints and improve generalization, this work replaces full-graph training with a minibatch sampling approach. Instead of processing the entire graph, the model trains on representative subsets of vertices.
Subgraph Sampling with ShaDow
GNNs typically suffer from "neighborhood explosion," where even a small batch of target vertices requires data from a massive, multi-hop neighborhood. To mitigate this, the authors utilize the ShaDow subgraph sampling algorithm. To ensure this process does not become a new bottleneck, two major performance optimizations were introduced:
- Matrix-Based Bulk Sampling: Traditional GNN sampling can consume up to 60% of total training time. The authors adapted the ShaDow algorithm to use matrix-based bulk sampling. By representing random walks and subgraph extraction as sparse matrix multiplications, the system can sample multiple minibatches simultaneously in a single step, significantly increasing GPU utilization.
- Coalesced All-Reduce: When utilizing Distributed Data Parallelism (DDP) across multiple GPUs, the model must synchronize gradients. The Interaction Network (IGNN) used in Exa.TrkX consists of multiple Multi-Layer Perceptrons (MLPs). Standard implementations trigger separate, high-latency
all-reducecalls for each MLP. The authors optimized this by stacking parameter matrices to execute a single, combinedall-reducecall, drastically reducing communication latency.
Experimental Setup
The optimized pipeline was evaluated on two standard HEP datasets:
- Connect the Dots (CTD): A massive dataset with graphs averaging 330.7K vertices and 6.9M edges. A representative sample of 80 graphs was used to demonstrate computational efficiency.
- Example 3 (Ex3): A smaller dataset with graphs averaging 13.0K vertices and 47.8K edges.
Results and Discussion
Accuracy and Convergence
The transition to minibatch training (using a batch size of 256) resulted in higher precision and recall compared to the baseline full-graph approach. This confirms that training on subsets of large graphs is more effective than skipping them. The stochastic noise inherent in small-batch training likely helped the model converge to sharper, more accurate minima.
Computational Performance
The optimized pipeline achieved a 1.3x to 2x speedup in epoch training time compared to standard PyTorch Geometric (PyG) implementations of the ShaDow algorithm. On the massive CTD dataset, the authors observed superlinear scaling: sampling time was approximately 27x faster when using four times the hardware. This occurs because matrix-based bulk sampling is memory-bound; increasing the number of GPUs expands the aggregate memory pool, allowing for significantly larger bulks of minibatches to be processed in parallel.
Conclusion
By shifting from full-graph training to an optimized minibatch sampling strategy, the Exa.TrkX pipeline can now process the largest HEP collision events without memory-related data loss. The use of matrix-based bulk sampling transforms a sequential, CPU-bound bottleneck into a parallelized GPU task, while coalesced communication reduces DDP overhead. While the speedups are most pronounced on larger datasets like CTD, the methodology provides a robust framework for scaling GNNs in other memory-intensive domains, such as social network analysis or molecular modeling. Future work may involve testing these dynamics on the full 7,800-graph CTD dataset to further explore long-term scaling dynamics.
Summary updated