Deep learning hardware, particularly GPUs and TPUs, is designed for dense, rectangular tensor operations. However, real-world data—ranging from natural language and audio to medical imaging and reinforcement learning trajectories—is inherently "ragged," consisting of sequences with widely varying lengths. The traditional solution is to pad shorter sequences with zeros to match the longest element in a batch. This approach incurs a significant "padding tax," where computational cycles and memory are wasted on processing null tokens.
Sequence packing addresses this inefficiency by concatenating multiple variable-length samples into a single, dense buffer. By treating a batch as a continuous stream of data rather than a collection of independent rows, researchers have achieved dramatic speedups in training and inference throughput across diverse modalities.
Eliminating the Padding Tax
The core intuition behind sequence packing is to maximize hardware occupancy. When sequences are padded, the GPU performs the same number of floating-point operations (FLOPs) on a zero as it does on a meaningful token. Xu et al. demonstrate in PackMamba that by packing variable-length sequences into fixed-size blocks, training throughput for state-space models can increase by over 3x. This is achieved by ensuring that every position in a hardware-optimized vector load contains actual data.
In the context of Vision Transformers, Dehghani et al. introduced "Patch n’ Pack" in NaViT. Rather than resizing images to a fixed resolution—which distorts aspect ratios or requires heavy padding—NaViT treats image patches as a sequence. By packing patches from multiple images of different resolutions into a single sequence, the model processes images at their native scale while maintaining near-perfect GPU utilization.
Optimization via Bucketing and Bin-Packing
Effective packing requires deciding which sequences should be grouped together to minimize residual space. Early implementations, such as those in Tensor2Tensor, utilized length-based bucketing. By grouping sequences of similar lengths, the authors minimized the amount of padding required within each batch.
More sophisticated approaches treat this as a variation of the bin-packing problem. Fang et al. proposed a dynamic programming-based batch scheduler in TurboTransformers to identify the optimal grouping of inference requests, significantly reducing the padding tax in real-time serving environments. Similarly, Nagarajan et al. introduced "BucketSampler" and residual batch consolidation to merge leftover samples from different buckets, ensuring that even the "ragged edges" of a dataset are processed efficiently.
Preventing Information Leakage
A primary challenge of packing is ensuring that independent sequences do not influence one another during the forward pass. In standard Transformers, this is managed through attention masking, which prevents tokens in one packed sequence from attending to tokens in another.
However, for models with recurrent or stateful components, the solution requires deeper architectural modifications. In PackMamba, the authors modified the core SSM and Conv1d operators to reset their internal states at the boundaries of packed sequences. This prevents information leakage, allowing the model to treat a single continuous tensor as a series of distinct logical units.
Scaling to Multimodal and Distributed Systems
As models grow to handle "any-to-any" modalities, sequence packing becomes essential for managing the vast disparity between short text snippets and high-resolution media. Lu et al. utilized dynamic packing in Unified-IO 2 to increase throughput by up to four times when training on mixtures of text, image, and audio data.
In large-scale distributed training, packing also helps mitigate communication bottlenecks. Yao et al. proposed Hierarchical Balance Packing (HBP) to address workload imbalances in long-context fine-tuning. By isolating shorter sequences into groups that do not require sequence parallelism, HBP eliminates unnecessary inter-GPU communication, achieving a 2.4x speedup on models as large as 236 billion parameters.
Beyond Efficiency: Improving Model Quality
While primarily an efficiency optimization, sequence packing can also improve learning dynamics. Pouransari et al. argue in Dataset Decomposition that training on variable-length sequences that respect actual document boundaries—rather than arbitrarily truncating them into fixed blocks—prevents cross-document contamination. This ensures the model learns from coherent context, leading to better performance on both standard and long-context benchmarks. Similarly, in the medical domain, Tang et al. showed that packing variable-length slide sequences in PackMIL not only slashed training time but also improved diagnostic accuracy by enabling higher-throughput processing of gigapixel-scale images.