Eliminating Pipeline Bubbles in Distributed Model Training

116 concepts96 papers1 proposal

When training or serving giant foundation models, a single GPU quickly runs out of memory. To scale these models, systems rely on 3D parallelism—a combination of data, tensor, and pipeline parallelism, as detailed by Lai et al.. In pipeline parallelism, a model's layers are sliced sequentially across a chain of GPUs.

However, this sequential partitioning introduces a fundamental inefficiency: the pipeline bubble. Because layer $L+1$ depends on the outputs of layer $L$, downstream GPUs sit completely idle during the startup phase of a forward pass, waiting for data to trickle down the chain. The same problem occurs in reverse during the teardown phase of the backward pass. As noted by Li et al., these synchronization gaps lead to massive hardware underutilization.

Why Bubbles Form: The Synchronous Bottleneck

To maintain exact mathematical equivalence to single-GPU training, standard synchronous pipeline schedules (such as 1F1B—One Forward, One Backward) require periodic flushes. As explained by Narayanan et al., these flushes mean that accelerators must finish their current batch before starting the next, creating predictable blocks of idle time.

The problem worsens under specific conditions:

  1. Long Contexts: As sequence lengths grow, the quadratic cost of attention expands execution time, causing bubbles to balloon (Zhang et al.).
  2. Heterogeneous Hardware: In mixed-GPU environments, faster devices are forced to stall while waiting for slower devices to synchronize (Qi).
  3. Sequence Length Variance: When batches contain sequences of varying lengths, GPUs processing shorter inputs waste cycles waiting for the longest sequence to finish (Yuan et al.).

To recover this wasted performance, researchers have converged on three main paradigms: filling bubbles with auxiliary work, restructuring the pipeline schedule, and changing the granularity of the data flow.

Paradigm 1: Exploiting the Bubble with Auxiliary Computation

If a GPU must wait for pipeline dependencies, why not give it a completely independent task to run in the meantime? This "bubble exploitation" strategy treats idle time as a free computational resource.

For instance, Osawa et al. developed PipeFisher, which fills pipeline bubbles with the heavy second-order optimization calculations required for Kronecker-Factored Approximate Curvature (K-FAC). Because these calculations are mathematically independent of the immediate forward/backward step, they can be computed during idle slots without delaying the main training loop.

Similarly, in multimodal and diffusion models, parts of the network are often frozen. Tian et al. (DiffusionPipe) and Lin et al. (Optimus) hide the processing of non-trainable components—such as frozen text encoders or image feature extractors—directly inside the pipeline gaps of the trainable layers.

This concept extends beyond model components:

  • Knowledge Distillation: Lim et al. (TSPipe) interleave forward-only teacher network computations into the student's pipeline bubbles.
  • Fault Tolerance: Park et al. (SpareTrain) schedule redundant "checker" tasks during these idle cycles to catch silent data corruption without incurring a throughput penalty.

Paradigm 2: Schedule Restructuring and Pass Splitting

Instead of filling the bubbles, can we eliminate them entirely by changing when operations occur?

A foundational approach by Narayanan et al. introduced an interleaved pipeline schedule (PTD-P), where each GPU holds multiple non-contiguous layers (e.g., GPU 0 holds layers 1 and 9), allowing it to switch to another layer's microbatch rather than sitting idle.

Taking this further, Qi proposed Zero Bubble Pipeline Parallelism. The core insight is that the backward pass consists of two distinct components: computing gradients with respect to the activations ($B_A$) and computing gradients with respect to the weights ($B_W$). Because $B_W$ is only needed before the optimizer step, its execution can be safely delayed and packed into the startup/teardown bubbles. This decouples the immediate dependency chain, reducing idle time to nearly zero.

Standard 1F1B Schedule:
GPU 0: [ F1 ][ F2 ][ B2 ][ B1 ] --(Bubble/Flush)--
GPU 1:   [ F1 ][ F2 ][ B2 ][ B1 ]

Zero-Bubble Schedule (Splitting Backward Pass):
GPU 0: [ F1 ][ F2 ][ BA2 ][ BA1 ][ BW2 ][ BW1 ]
GPU 1:   [ F1 ][ F2 ][ BA2 ][ BA1 ][ BW2 ][ BW1 ]

Other structural innovations include the "wave-like" schedules proposed by Cheng et al. (Hanayo), bidirectional pipelines by Li et al. (Chimera), and automated scheduling frameworks like OptPipe by Li et al., which uses Mixed-Integer Linear Programming to mathematically minimize bubble overhead based on specific hardware profiles.

Paradigm 3: Fine-Grained and Token-Level Pipelining

Traditional pipeline parallelism operates at the level of the microbatch. If a microbatch is large, the time a downstream GPU spends waiting for the entire chunk to process is correspondingly large.

To shrink the bubble, systems can pipeline along finer dimensions. Li et al. introduced TeraPipe, which pipelines along the token dimension within a single sequence. By exploiting the autoregressive structure of Transformers, downstream layers can begin processing the first few tokens of a sequence while upstream layers are still computing subsequent tokens.

For long-context fine-tuning, Sun et al. developed Seq1F1B, scheduling at the sequence level to exploit causal attention masks. In inference serving, Agrawal et al. (Sarathi-Serve) solve generation stalls by chunking compute-heavy prefill requests and "piggybacking" them onto memory-bound decode iterations, smoothing out execution variances that would otherwise create massive pipeline bubbles.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 97 papers
  2. Version 1 · 2026-10-04 · 201 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_pipeline_bubbles_ad2ff866,
  title = {pipeline bubbles},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/pipeline-bubbles/rdr_ad2ff866acb143159dd9cd2295c63e9d},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 2, frozen text):

@misc{lacuna_pipeline_bubbles_ad2ff866_v2,
  title = {pipeline bubbles},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/pipeline-bubbles/rdr_ad2ff866acb143159dd9cd2295c63e9d/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake