Standard deep learning relies on backpropagation, which imposes a strict sequential dependency across a network's layers. This dependency manifests as two distinct bottlenecks: backward locking (layer $i$ cannot compute its activation gradients until layer $i+1$ has finished doing so) and update locking (weights cannot be updated until the global backward pass reaches that layer).
When training massive models across distributed hardware or pipeline-parallel setups, these locks force expensive accelerators to sit idle, creating "pipeline bubbles." To maximize hardware utilization, a growing body of research focuses on decoupling the layers—allowing them to compute, pass messages, and update parameters asynchronously.
Splitting the Backward Pass: Separating Activations and Weights
To understand how we can break the sequential lock of backpropagation, we must look closely at what a layer actually does during the backward pass. At any given layer, backpropagation computes two distinct sets of gradients:
- Activation gradients ($\nabla_a$): The error signal with respect to the layer's inputs, which must be passed backward to the preceding layer.
- Weight gradients ($\nabla_w$): The error signal with respect to the layer's own parameters, used to update its weights.
Crucially, while $\nabla_a$ has an immediate downstream dependency (the preceding layer cannot start without it), $\nabla_w$ does not. Once a layer receives its error signal and computes $\nabla_a$, it can hand that signal off immediately. The computation of $\nabla_w$ and the subsequent weight update can happen whenever convenient.
Penghui Qi et al. exploit this asymmetry in Zero Bubble (Almost) Pipeline Parallelism. By splitting the backward pass into these two independent operations, their scheduling strategy deprioritizes the weight gradient calculation, using it as a "filler" task to occupy GPUs during periods that would otherwise be pipeline bubbles.
As formalized by Hongpei Li et al. in OptPipe, treating activation updates and weight updates as decoupled tasks transforms pipeline scheduling into a flexible optimization problem, allowing systems to hide weight update latency entirely within structural gaps.
Standard Backward Pass:
[Layer 3: Compute ∇a & ∇w] ──> [Layer 2: Compute ∇a & ∇w] ──> [Layer 1: Compute ∇a & ∇w]
Decoupled Backward Pass:
[Layer 3: Compute ∇a] ─────────> [Layer 2: Compute ∇a] ─────────> [Layer 1: Compute ∇a]
│ │ │
└──> [Layer 3: Compute ∇w] └──> [Layer 2: Compute ∇w] └──> [Layer 1: Compute ∇w]
(Scheduled flexibly) (Scheduled flexibly) (Scheduled flexibly)
Embracing Staleness: Asynchronous Queues and Buffers
If we want to decouple layers even further, we must address backward locking directly: can a layer proceed with its forward or backward pass using slightly outdated information from its neighbors?
Allowing "staleness" breaks the synchronization barriers of backpropagation. An Xu et al. introduce this mechanism in On the Acceleration of Deep Learning Model Parallelism With Staleness through a framework called Diversely Stale Parameters (DSP). By placing First-In-First-Out (FIFO) queues between layers, each layer can read activations from preceding layers and gradients from succeeding layers asynchronously. The layers no longer lock step; they simply process the latest available data in their local buffers.
This concept is extended to sequence modeling by Qian Yang et al. in Ouroboros, which uses delayed gradients from previous training iterations to allow multiple pipeline stages to execute concurrently.
To prevent the shifting distribution of these stale activations from destabilizing training, Eugene Belilovsky et al. utilize replay buffers between network modules in Decoupled Greedy Learning of CNNs. The replay buffer stores historical activations, allowing subsequent layers to continue training smoothly even when their predecessors are delayed or updating at different rates.
Bypassing the Chain: Local Losses and Direct Feedback
The most radical way to decouple layers is to eliminate the global backpropagation chain entirely. If a layer does not rely on a signal propagated sequentially from the final output, it can update its parameters completely in parallel with every other layer.
One prominent strategy is Greedy Block-wise Learning, which breaks the network into independent modules, each equipped with its own small, auxiliary local classifier or loss function. Myeongjang Pyeon et al. automate the discovery of these architectures in SEDONA, demonstrating that local objectives can successfully eliminate backward locking.
This approach is particularly powerful for distributed edge-cloud systems. As shown by Pedram Pad et al. in Hierarchical Training of Deep Neural Networks Using Early Exiting, repurposing early-exit classifiers allows an edge device to compute local losses and update its lower-layer parameters immediately, removing the need to wait for a global gradient return from a cloud server.
Global Backpropagation:
Input ──> [Layer 1] ──> [Layer 2] ──> [Layer 3] ──> Global Loss
│ │ │ │
└<─── ∇a ────┴<─── ∇a ────┴<───── ∇a ─────┘ (Sequential Chain)
Greedy Local Learning:
Input ──> [Layer 1] ──> [Layer 2] ──> [Layer 3]
│ │ │
├──> Local ├──> Local └──> Global Loss
│ Loss 1 │ Loss 2 │
└──> [Update]└──> [Update] └──> [Update] (Parallel Updates)
Another alternative is Direct Feedback Alignment (DFA), which bypasses the layer-by-layer chain by projecting the global error signal directly to each hidden layer using fixed, random matrices. Justin Gilmer et al. demystify this phenomenon in Explaining the Learning Dynamics of Direct Feedback Alignment, showing that DFA mathematically behaves like a noisy version of layer-wise supervised training where each layer solves the task using a random local classifier.
Because the error signal is broadcast directly rather than backpropagated through intervening layers, the sequential dependency is broken. The author Julien Launay demonstrates in Direct Feedback Alignment Scales to Modern Deep Learning Tasks and Architectures that this direct signaling allows for asynchronous layer updates across modern, large-scale architectures like Transformers.
Forward-Only and Reversible Flows
Decoupling can also be achieved by changing how data and gradients flow through time. In standard backpropagation, hardware must stash all intermediate activations during the forward pass because they are required later to compute gradients during the backward pass. This creates a massive memory footprint and locks the forward and backward phases into a strict sequence.
Stephane Rivaud et al. address this memory-induced sequencing in PETRA: Parallel End-to-end Training with Reversible Architectures. By utilizing reversible neural layers, each stage of a distributed model can reconstruct its required activations on the fly from downstream outputs. This allows hardware stages to execute forward and backward passes concurrently, processing backward signals the moment they arrive without waiting for a global forward pass to conclude.
Taking this concept to its logical conclusion, the authors of Gradient Forward-Propagation for Large-Scale Temporal Video Modelling eliminate the backward pass entirely by propagating error signals forward alongside activations. By decoupling the layers such that each resides on an independent device and updates its parameters in real-time as data streams past, the system throughput becomes bounded only by the latency of the slowest individual layer, rather than the cumulative time of a full global loop.