BLoad: Enhancing Neural Network Training with Efficient Sequential Data Handling

Raphael Ruschel, A. S. M. Iftekhar, B. S. Manjunath, Suya YouarXiv 2023 · OpenReview

Abstract

The increasing complexity of modern deep neural network models and the expanding sizes of datasets necessitate the development of optimized and scalable training methods. In this white paper, we addressed the challenge of efficiently training neural network models using sequences of varying sizes. To address this challenge, we propose a novel training scheme that enables efficient distributed data-parallel training on sequences of different sizes with minimal overhead. By using this scheme we were able to reduce the padding amount by more than 100x while not deleting a single frame, resulting in an overall increased performance on both training time and Recall in our experiments.

Introduction: The Synchronization Bottleneck

Training deep learning models on sequences of varying lengths—such as video clips ranging from 2 to 10 seconds—presents a significant challenge for Distributed Data-Parallel (DDP) frameworks. The primary bottleneck lies in gradient synchronization. In a DDP setup, multiple GPUs process separate batches and must synchronize their gradients before updating the model.

When GPUs are assigned batches with different total sequence lengths, a "stalling" problem occurs. If GPU 1 processes a batch of short sequences while GPU 2 processes long ones, GPU 1 finishes its computation early and remains idle. In many cases, this leads to a complete deadlock where the system stalls indefinitely because the synchronization step expects simultaneous data from all nodes.

Distributed training often stalls when GPUs process batches of unequal lengths, leading to deadlocks during gradient synchronization.

Existing solutions typically rely on naive padding (padding all sequences to the maximum length) or chunking/sampling (trimming sequences). Naive padding is computationally wasteful, often processing more "zero" frames than actual data. Conversely, sampling discards valuable information and destroys long-term temporal dependencies.

The BLoad Methodology

To overcome these limitations, we introduce BLoad (Block Load), a strategy that optimizes padding by packing multiple sequences into fixed-size blocks.

Sequence Concatenation and Minimal Padding

Instead of padding each individual sequence to a global maximum length ($T_{max}$), BLoad randomly samples multiple sequences and concatenates them to fill a block of size $T_{max}$. Padding is only applied to the small remaining gap if the sequences do not perfectly fit the block. This approach ensures that every GPU processes a block of identical size, maintaining perfect synchronization in DDP without the overhead of massive padding.

BLoad packs multiple variable-length sequences into uniform blocks (BL1-BL4) across GPUs to ensure balanced workloads and efficient synchronization.

Metadata Tracking and State Management

A critical component of BLoad is the maintenance of a "starting indices" table. For models with temporal dependencies (like recurrent neural networks), it is essential to know where one sequence ends and another begins. This metadata allows the model to reset its internal hidden states at the correct frame, preventing information "bleeding" between unrelated sequences.

Experimental Evaluation

We evaluated BLoad using the Action Genome dataset, which consists of 7,464 videos with lengths ranging from 3 to 94 frames. The task involved Scene-Graph Detection using the Decoupled Dynamic Scene-graph (DDS) architecture.

Key Results

The comparison between BLoad, naive padding, and sampling strategies demonstrates significant improvements in both efficiency and accuracy.

Metric Naive Padding Sampling BLoad (Block Pad)
Padding Frames 534,831 0 3,695 (~144x less)
Frames Deleted 0 92,271 0
Time per Epoch 170 min 18 min 41 min
Recall@20 N/A (Too slow) 41.2 43.3

BLoad reduced the number of padding frames by approximately 144x compared to naive padding. While the sampling method is faster, it required deleting over 92,000 frames, which negatively impacted model performance. BLoad achieved a higher Recall@20 (43.3) by utilizing the full dataset without the prohibitive time cost of naive padding.

Conclusion

BLoad addresses the "stalling" problem in distributed training by treating the training batch as a container to be packed efficiently. By reducing padding overhead and maintaining data integrity, BLoad offers a scalable solution for training on variable-length sequential data. Future work will explore the application of BLoad to other modalities, such as audio and text, and its performance across diverse architectures like Transformers.

Summary updated

Abstract

The increasing complexity of modern deep neural network models and the expanding sizes of datasets necessitate the development of optimized and scalable training methods. In this white paper, we addressed the challenge of efficiently training neural network models using sequences of varying sizes. To address this challenge, we propose a novel training scheme that enables efficient distributed data-parallel training on sequences of different sizes with minimal overhead. By using this scheme we were able to reduce the padding amount by more than 100$x$ while not deleting a single frame, resulting in an overall increased performance on both training time and Recall in our experiments.
BibTeX
@misc{ruschel2023bload,
  author = {Raphael Ruschel and A. S. M. Iftekhar and B. S. Manjunath and Suya You},
  title = {{BLoad: Enhancing Neural Network Training with Efficient Sequential Data Handling}},
  year = {2023},
  eprint = {2310.10879},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2310.10879},
}
Figure