Training deep neural networks is notoriously expensive, primarily because of a fundamental symmetry in standard backpropagation: for every forward pass through a layer, there is a corresponding backward pass to compute gradients. This symmetry means that as networks grow deeper, the computational and memory costs scale linearly with the number of layers.
However, during transfer learning, fine-tuning, or the later stages of training, this full-network symmetry is highly redundant. Deep networks learn hierarchically; early layers often capture general, stable features that change very little, while later layers or specialized modules adapt to the specific task. By breaking the backpropagation chain early—a concept known as partial or selective backpropagation—and pairing it with activation caching, we can bypass massive blocks of computation entirely.
The Asymmetry of Training Bottlenecks
In a standard training loop, the backward pass is significantly more demanding than the forward pass. It requires calculating gradients for both the activations and the weights, and it forces the system to keep massive activation tensors in memory until the error signals flow all the way back to the input layer.
As demonstrated by Yue Wang in E2-Train: Training State-of-the-art CNNs with Over 80% Energy Savings, the backward pass is the primary bottleneck for energy-efficient and on-device learning. By shifting our focus from optimizing the forward pass to dynamically truncating the backward pass, we can unlock much larger efficiency gains.
Truncating the Gradient Flow
The simplest way to exploit this bottleneck is to freeze the lower layers of a pre-trained backbone model. When a layer is frozen, its parameters do not change, meaning we do not need to compute gradients for its weights. More importantly, we do not need to propagate the error gradient through it at all. The chain rule can simply stop at the lowest trainable layer.
Standard Backprop:
[Input] ---> [Layer 1] ---> [Layer 2] ---> [Layer 3] ---> [Loss]
| | | | |
[Grad] <--- [Grad W1] <--- [Grad W2] <--- [Grad W3] <-------+
Partial Backprop (Layer 1 & 2 Frozen):
[Input] ---> [Layer 1] ---> [Layer 2] ---> [Layer 3] ---> [Loss]
| |
[Grad W3] <------+
The authors of AdapterDrop: On the Efficiency of Adapters in Transformers (Andreas Rücklé et al.) leveraged this mechanism by freezing a transformer backbone and updating only higher-level adapter modules. By terminating backpropagation at the lowest trainable adapter layer, they achieved training speeds up to 60% faster than full fine-tuning.
Similarly, Tianxiang Sun et al. introduced Late Prompt Tuning: A Late Prompt Could Be Better Than Many Prompts, which inserts trainable parameters exclusively into the middle or upper layers of a network. Because all preceding layers are frozen, the backpropagation path is truncated early, drastically reducing memory consumption and gradient calculation overhead.
Eliminating Redundant Forwards via Activation Caching
While partial backpropagation eliminates the backward pass through frozen layers, a naive implementation still executes the forward pass through those same layers during every single epoch. If the weights of the early layers are frozen, and the input dataset is static, the output of those frozen layers will be identical every time a sample is processed. Running the forward pass repeatedly is entirely redundant.
Activation caching solves this by running the forward pass through the frozen layers exactly once (typically during the first epoch). The intermediate hidden states—the activations at the boundary where freezing ends and training begins—are saved to memory or disk. In all subsequent epochs, the training loop bypasses the frozen layers entirely, loading the cached activations directly and feeding them straight into the trainable layers.
Epoch 1 (Cache Generation):
[Input] ---> [Frozen Layers] ---> (Save Activation Cache) ---> [Trainable Layers] ---> [Loss]
Epoch 2+ (Bypassing the Backbone):
[Loaded Cache] ----------> [Trainable Layers] ---> [Loss]
This combination of partial backpropagation and activation caching transforms a deep training problem into a shallow one. For instance, Hiroki Matsutani et al. developed Skip2-LoRA: A Lightweight On-device DNN Fine-tuning Method for Low-cost Edge Devices, which caches intermediate activations of early layers during the initial training epoch. On resource-constrained edge devices like a Raspberry Pi, this optimization slashes fine-tuning time by up to 90% because subsequent epochs focus compute resources solely on the lightweight adapters.
This paradigm is equally powerful when multi-task learning occurs over a static corpus. As shown by Jon Saad-Falcon et al. in Embedding Recycling for Language Models, reusing stored intermediate representations across multiple downstream tasks yields massive speedups in both training and inference, capturing over 85% efficiency gains with negligible loss in accuracy.
Even in distributed environments, the strategy holds. Chaoyang He et al. developed PipeTransformer: Automated Elastic Pipelining for Distributed Training of Large-scale Models, which freezes layers as they stabilize and caches their output activations in a shared memory space. This allows newly spawned model replicas to immediately utilize pre-calculated features without repeating the initial layers' computations.
Adaptive and Multi-Rate Freezing
We do not have to decide which layers to freeze ahead of time. Instead, we can let the network dynamically determine which components require updates based on their convergence behavior or trainability.
- Granular Neuron Freezing: In SCoTTi: Save Computation at Training Time with an adaptive framework, Enzo Tartaglione et al. proposed treating the freezing threshold as a learnable parameter. Instead of freezing entire layers, the framework skips gradient calculations and weight updates for individual neurons whose activity has stabilized below a specific threshold, cutting down floating-point operations dynamically.
- Multi-Rate Updates: Rather than completely freezing a backbone for the duration of training, we can update different parts of the network at different frequencies. In Multirate Training of Neural Networks, Tiffany J. Vlaar demonstrated that updating task-specific layers at every step while updating the pre-trained backbone only once every several steps cuts training time nearly in half. This decoupled approach reduces the total number of expensive backpropagation passes through the entire network while preserving the model's capacity to adapt.
By recognizing that not all parameters learn at the same rate—and that frozen parameters generate deterministic outputs—partial backpropagation and activation caching successfully dismantle the traditional, symmetric bottlenecks of deep network training.
Key Foundational Work
Transfer Learning for Speech Recognition on a Budget (2017) — contributed 1 concept elements to this direction.
CRF-OPT: An Efficient High-Quality Conditional Random Field Solver (2008) — contributed 1 concept elements to this direction.