Scaling Distributed Training of Flood-Filling Networks on HPC Infrastructure for Brain Mapping

Wushi Dong, Murat Keçeli, Rafael Vescovi, Hanyu Li, Corey Adams, Thomas D. Uram, Venkatram Vishwanath, Bobby Kasthuri, Nicola J. Ferrier, Peter Littlewood

Introduction

Mapping the "connectome"—the intricate web of neural connections within the brain—represents one of the most significant data challenges in modern neuroscience. To map just 1 $cm^3$ of brain tissue at nanometer resolution requires approximately one petabyte of Electron Microscopy (EM) data. While Flood-Filling Networks (FFNs) are the current gold standard for segmenting these massive 3D image volumes into individual neurons, they are notoriously computationally expensive to train.

Historically, FFN implementations relied on asynchronous training schemes that struggled to scale effectively on high-performance computing (HPC) clusters. Researchers from Argonne National Laboratory and the University of Chicago have addressed this bottleneck by successfully scaling FFN training to 2,048 nodes on the Theta supercomputer. By transitioning to synchronous data-parallel training, they reduced training times from weeks to just four hours while maintaining high segmentation accuracy.

Methods and Architecture

The core of the approach involves moving away from the original asynchronous parameter server model in favor of synchronous data-parallel distributed training utilizing the Horovod library.

The FFN architecture consists of a 12-module recurrent 3D convolutional neural network that iteratively "fills" neuronal shapes.

Key technical implementations include:

  • Recurrent 3D CNN: The model uses a 12-module recurrent architecture to iteratively predict the shape of a neuron within a field of view (FOV).
  • Communication Efficiency: A ring-allreduce algorithm (via MPI) was employed to average gradients across nodes, significantly reducing communication overhead compared to traditional parameter servers.
  • Optimization Strategy: The team utilized the Adam optimizer and implemented a hybrid scaling policy for learning rates—linear scaling for smaller batches and square root scaling for very large batches—to prevent model divergence at scale.
  • HPC Tuning: Performance on Intel Knights Landing (KNL) processors was optimized by balancing MPI ranks and OpenMP threads to maximize core utilization.

Datasets

The study utilized the FIB-25 fly brain dataset, a publicly available volume of a Drosophila (fruit fly) medulla acquired via Focused Ion Beam Scanning Electron Microscopy (FIB-SEM). The training was performed on a $520^3$-voxel subvolume, with a separate $250^3$-voxel subvolume used for validation. The 8x8x8 nm resolution provides the necessary detail to trace individual neuronal processes.

Performance and Scaling Insights

The transition to synchronous training revealed several critical insights into the behavior of FFNs at scale. The researchers discovered that on a single node, increasing the number of MPI ranks (processes) was more effective for throughput than simply increasing the batch size within a single rank. This approach improved load balancing across the KNL processor's 64 cores.

Throughput analysis shows that increasing the number of ranks is more effective for performance than increasing batch size alone.

Furthermore, while increasing the number of nodes drastically reduces "wall-clock" training time, there is a trade-off with "training efficiency" (the contribution of each FOV to model improvement). The team identified an optimal batch size that balances throughput with convergence speed.

Key Results

  • Massive Speedup: The time required to reach 95% accuracy was reduced to approximately 4 hours, a significant improvement over the days or weeks required by previous implementations.
  • Near-Ideal Scaling: The system demonstrated nearly linear scaling up to 2,048 KNL nodes on the Theta supercomputer.
  • High Throughput: At peak scale, the system achieved a sustained throughput of approximately 1,008 FOVs/second with a parallel efficiency of about 68%.
  • Accuracy Preservation: Despite the change in training methodology, the resulting models achieved Adapted Rand Error (ARE) and Variation of Information (VOI) scores comparable to the original implementation.

Limitations and Future Work

A primary challenge in synchronous SGD is the "straggler" problem, where the slowest node dictates the speed of the entire cluster. Future iterations may incorporate backup workers to mitigate this issue. Additionally, while training has been successfully scaled, integrating this framework into the full pipeline—including data pre-processing and post-segmentation agglomeration—remains a key goal for mapping entire mammalian brains.

Summary updated

BibTeX
@inproceedings{dong2019scaling,
  author = {Wushi Dong and Murat Keçeli and Rafael Vescovi and Hanyu Li and Corey Adams and Thomas D. Uram and Venkatram Vishwanath and Bobby Kasthuri and Nicola J. Ferrier and Peter Littlewood},
  title = {{Scaling Distributed Training of Flood-Filling Networks on HPC Infrastructure for Brain Mapping}},
  booktitle = {DLS@SC},
  year = {2019},
  doi = {10.1109/dls49591.2019.00012},
}
Figure