Abstract
The increasing complexity of modern deep neural network models and the expanding sizes of datasets necessitate the development of optimized and scalable training methods. In this white paper, we addressed the challenge of efficiently training neural network models using sequences of varying sizes. To address this challenge, we propose a novel training scheme that enables efficient distributed data-parallel training on sequences of different sizes with minimal overhead. By using this scheme we were able to reduce the padding amount by more than 100x while not deleting a single frame, resulting in an overall increased performance on both training time and Recall in our experiments.
Introduction: The Synchronization Bottleneck
Training deep learning models on sequences of varying lengths—such as video clips ranging from 2 to 10 seconds—presents a significant challenge for Distributed Data-Parallel (DDP) frameworks. The primary bottleneck lies in gradient synchronization. In a DDP setup, multiple GPUs process separate batches and must synchronize their gradients before updating the model.
When GPUs are assigned batches with different total sequence lengths, a "stalling" problem occurs. If GPU 1 processes a batch of short sequences while GPU 2 processes long ones, GPU 1 finishes its computation early and remains idle. In many cases, this leads to a complete deadlock where the system stalls indefinitely because the synchronization step expects simultaneous data from all nodes.
Existing solutions typically rely on naive padding (padding all sequences to the maximum length) or chunking/sampling (trimming sequences). Naive padding is computationally wasteful, often processing more "zero" frames than actual data. Conversely, sampling discards valuable information and destroys long-term temporal dependencies.
The BLoad Methodology
To overcome these limitations, we introduce BLoad (Block Load), a strategy that optimizes padding by packing multiple sequences into fixed-size blocks.
Sequence Concatenation and Minimal Padding
Instead of padding each individual sequence to a global maximum length ($T_{max}$), BLoad randomly samples multiple sequences and concatenates them to fill a block of size $T_{max}$. Padding is only applied to the small remaining gap if the sequences do not perfectly fit the block. This approach ensures that every GPU processes a block of identical size, maintaining perfect synchronization in DDP without the overhead of massive padding.
Metadata Tracking and State Management
A critical component of BLoad is the maintenance of a "starting indices" table. For models with temporal dependencies (like recurrent neural networks), it is essential to know where one sequence ends and another begins. This metadata allows the model to reset its internal hidden states at the correct frame, preventing information "bleeding" between unrelated sequences.
Experimental Evaluation
We evaluated BLoad using the Action Genome dataset, which consists of 7,464 videos with lengths ranging from 3 to 94 frames. The task involved Scene-Graph Detection using the Decoupled Dynamic Scene-graph (DDS) architecture.
Key Results
The comparison between BLoad, naive padding, and sampling strategies demonstrates significant improvements in both efficiency and accuracy.
| Metric | Naive Padding | Sampling | BLoad (Block Pad) |
|---|---|---|---|
| Padding Frames | 534,831 | 0 | 3,695 (~144x less) |
| Frames Deleted | 0 | 92,271 | 0 |
| Time per Epoch | 170 min | 18 min | 41 min |
| Recall@20 | N/A (Too slow) | 41.2 | 43.3 |
BLoad reduced the number of padding frames by approximately 144x compared to naive padding. While the sampling method is faster, it required deleting over 92,000 frames, which negatively impacted model performance. BLoad achieved a higher Recall@20 (43.3) by utilizing the full dataset without the prohibitive time cost of naive padding.
Conclusion
BLoad addresses the "stalling" problem in distributed training by treating the training batch as a container to be packed efficiently. By reducing padding overhead and maintaining data integrity, BLoad offers a scalable solution for training on variable-length sequential data. Future work will explore the application of BLoad to other modalities, such as audio and text, and its performance across diverse architectures like Transformers.
Summary updated