To train deep learning models faster, the most common strategy is to increase the batch size and distribute the workload across hundreds or thousands of processors. However, this approach traditionally hits a "wall": as the batch size grows, the model often fails to converge or suffers a significant drop in accuracy.
The breakthrough that enabled training ResNet-50 in minutes rather than days was the realization that optimization instability is not uniform across a network. Instead, different layers require different "speed limits" to remain stable. This led to the development of Layer-wise Adaptive Rate Scaling (LARS) and its successors, which allow batch sizes to scale to 32,000 and beyond.
The Problem of Global Learning Rates
When training with massive batches, researchers typically use the "linear scaling rule," where the learning rate is increased proportionally to the batch size to maintain the same number of updates per epoch. However, Igor Gitman and Boris Ginsburg observed in Large Batch Training of Convolutional Networks that this global scaling is too aggressive for certain layers.
In a deep network, the ratio between the norm of the weights ($\|w\|$) and the norm of the gradients ($\|\nabla w\|$) varies wildly between layers. If a layer has very small weights but receives a large gradient update, it can become unstable and cause the entire model to diverge. Conversely, if the update is too small relative to the weights, the layer stalls. A single global learning rate cannot satisfy these conflicting requirements.
The Trust Ratio
The core innovation of LARS is the Trust Ratio. Instead of a single learning rate for the whole model, LARS calculates a local learning rate for each layer $l$:
$$\lambda^l = \eta \cdot \frac{\|w^l\|}{\|\nabla w^l\| + \beta \|\nabla w^l\|}$$
where $\eta$ is a global scaling factor. This mechanism ensures that the update magnitude is always proportional to the magnitude of the weights in that specific layer.
By using this layer-wise approach, Yang You and his colleagues demonstrated in ImageNet Training in Minutes that they could scale ImageNet training to a batch size of 32,000, reducing training time for ResNet-50 to just 14 minutes without losing accuracy.
Preserving Geometry vs. Element-wise Adaptation
A natural question is why we need layer-wise scaling if we already have element-wise adaptive optimizers like Adam. The evidence suggests that "granularity" matters.
As noted in Scaling SGD Batch Size to 32K for ImageNet Training, element-wise optimizers adjust the rate for every individual weight, which can distort the direction of the gradient and harm generalization. Layer-wise adaptation, however, uses a single scalar multiplier for an entire weight tensor. This preserves the global geometry of the optimization landscape—the direction of the update remains the same as the original gradient, but its magnitude is "tamed" to fit the layer's scale.
Scaling Beyond Vision: LAMB and BERT
The success of LARS in computer vision inspired the LAMB (Layer-wise Adaptive Moments optimizer for Batch training) optimizer. In Large Batch Optimization for Deep Learning: Training BERT in 76 minutes, Yang You and Sashank J. Reddi combined the layer-wise trust ratio of LARS with the adaptive moment estimation of Adam.
LAMB enabled the scaling of BERT training to a batch size of 64,000. By normalizing updates based on the ratio of weight norms to update norms, LAMB prevents the gradient explosions common in Transformer architectures at high learning rates. This reduced BERT's training time from three days to just over an hour.
Applications in Alternative Architectures
The principle of layer-wise scaling has proven robust across diverse research areas where gradient imbalance is a primary bottleneck:
- AdderNets: In AdderNet: Do We Really Need Multiplications in Deep Learning?, Hanting Chen and Yunhe Wang used layer-wise scaling to compensate for the high variance of additive filters, which lack the natural normalization of standard convolutions.
- Predictive Coding: Biologically inspired alternatives to backpropagation often suffer from vanishing updates in early layers. Nick Alonso and Emre Neftci showed in Understanding and Improving Optimization in Predictive Coding Networks that equalizing update magnitudes with a single scalar per layer provides stability comparable to Adam but with a much lower memory footprint.
- Federated Learning: When training on decentralized data, Belhal Karimi and Xiaoyun Li found that layer-wise normalization helps models converge faster in non-IID environments by ensuring that layers with different gradient scales evolve at a compatible pace (Fed-LAMB).
Theoretical Grounding
While early layer-wise methods were largely empirical, recent work has sought to provide a mathematical foundation. Zhouyuan Huo and Heng Huang introduced CLARS (Large Batch Optimization for Deep Learning Using New Complete Layer-Wise Adaptive Rate Scaling), arguing that layer-wise scaling effectively approximates scaling by the inverse of the layer-specific Lipschitz constant. This suggests that the "trust ratio" is essentially a way of automatically flattening the optimization landscape, making it more amenable to the aggressive steps required by large-batch training.