Robust Training of Neural Networks using Scale Invariant Architectures

Zhiyuan Li, Srinadh Bhojanapalli, Manzil Zaheer, Sashank J. Reddi, Sanjiv Kumar

Abstract

In contrast to SGD, adaptive gradient methods like Adam allow robust training of modern deep networks, especially large language models. However, the use of adaptivity not only comes at the cost of extra memory but also raises the fundamental question: can non-adaptive methods like SGD enjoy similar benefits? In this paper, we provide an affirmative answer to this question by proposing to achieve both robust and memory-efficient training via the following general recipe: (1) modify the architecture and make it scale invariant, i.e. the scale of parameter doesn't affect the output of the network, (2) train with SGD and weight decay, and optionally (3) clip the global gradient norm proportional to weight norm multiplied by $\sqrt{\tfrac{2\lambda}{\eta}}$, where $\eta$ is learning rate and $\lambda$ is weight decay. We show that this general approach is robust to rescaling of parameter and loss by proving that its convergence only depends logarithmically on the scale of initialization and loss, whereas the standard SGD might not even converge for many initializations. Following our recipe, we design a scale invariant version of BERT, called SIBERT, which when trained simply by vanilla SGD achieves performance comparable to BERT trained by adaptive methods like Adam on downstream tasks.

Introduction

Modern deep learning architectures, particularly Transformers, are notoriously difficult to optimize. To maintain stability and prevent training loss from diverging, practitioners rely heavily on adaptive optimizers such as Adam, AdaGrad, or LAMB. However, these methods come with a significant memory overhead: Adam requires storing first and second moments of gradients, consuming approximately three times more memory for optimizer states than vanilla Stochastic Gradient Descent (SGD).

This paper investigates whether the stability of adaptive methods can be achieved using the memory efficiency of SGD. The authors hypothesize that the instability of standard networks stems from their $k$-homogeneous nature, where scaling parameters by $s$ scales the output by $s^k$. This creates a positive feedback loop where large updates increase parameter norms, which in turn scale up gradients, leading to explosive instability. To solve this, the authors propose a "scale-invariant" approach.

Methodology: The Scale-Invariant Recipe

The proposed framework consists of three primary components designed to stabilize SGD training by mathematically smoothing the loss landscape.

1. Scale Invariant Architecture (SIBERT)

The authors redesign the BERT architecture to be 0-homogeneous (scale-invariant), meaning the network's output remains unchanged regardless of parameter scaling. A key modification is the introduction of Scale Invariant Attention. Standard Softmax is not homogeneous; the authors replace it with a ReLU activation followed by row-wise sum normalization to maintain scale invariance.

2. SGD with Weight Decay (WD)

In a scale-invariant network, the effective learning rate is inversely proportional to the square of the parameter norm. The authors prove that SGD with Weight Decay acts as an auto-tuning mechanism for this effective learning rate. If gradients become too small, weight decay reduces the parameter norm, which naturally increases the effective learning rate to maintain training momentum.

3. Relative Global Clipping

Traditional gradient clipping relies on a fixed threshold that is difficult to tune. The authors introduce a dynamic clipping rule where the threshold is proportional to the current parameter norm ($\lVert x \rVert_2$) and the square root of the ratio between weight decay ($\lambda$) and learning rate ($\eta$):

$$\text{Threshold} \propto \lVert x \rVert_2 \sqrt{\frac{2\lambda}{\eta}}$$

This prevents sudden loss spikes without introducing the bias common in fixed-threshold clipping.

Experimental Results

The authors evaluated SIBERT on standard pretraining tasks and downstream benchmarks including SQuAD v1.1, SQuAD v2.0, and MNLI.

  • Memory Efficiency: By utilizing vanilla SGD (even without momentum), the optimizer memory footprint was reduced to a fraction of that required by Adam.
  • Competitive Performance: SIBERT trained with the SGD+WD+Clipping recipe achieved performance nearly identical to standard BERT. On SQuAD v1.1, SIBERT reached an 89.3 F1 score (vs. 90.3 for BERT). With an extended training budget, SIBERT matched the baseline exactly at 90.3 F1.
  • Extreme Robustness: The recipe demonstrated remarkable stability across initialization scales spanning two orders of magnitude. While standard SGD diverged when initialization scales were mismatched, the SIBERT recipe converged identically regardless of the starting scale.

Discussion and Conclusion

The findings suggest that the need for complex adaptive optimizers can be bypassed by co-designing the neural architecture and the optimization dynamics. In scale-invariant networks, weight decay transitions from a simple regularizer to an active steering mechanism for optimization.

While SIBERT achieves parity with Adam-trained models, some limitations remain. The classification head of SIBERT was not made scale-invariant in this study, and a slight performance gap exists unless the training budget is increased. Future work in "architecture-aware optimization" may further bridge these gaps, leading to even more efficient and robust training paradigms for large-scale models.

Summary updated

Abstract

In contrast to SGD, adaptive gradient methods like Adam allow robust training of modern deep networks, especially large language models. However, the use of adaptivity not only comes at the cost of extra memory but also raises the fundamental question: can non-adaptive methods like SGD enjoy similar benefits? In this paper, we provide an affirmative answer to this question by proposing to achieve both robust and memory-efficient training via the following general recipe: (1) modify the architecture and make it scale invariant, i.e. the scale of parameter doesn't affect the output of the network, (2) train with SGD and weight decay, and optionally (3) clip the global gradient norm proportional to weight norm multiplied by $\sqrt{\tfrac{2\lambda}{\eta}}$, where $\eta$ is learning rate and $\lambda$ is weight decay. We show that this general approach is robust to rescaling of parameter and loss by proving that its convergence only depends logarithmically on the scale of initialization and loss, whereas the standard SGD might not even converge for many initializations. Following our recipe, we design a scale invariant version of BERT, called SIBERT, which when trained simply by vanilla SGD achieves performance comparable to BERT trained by adaptive methods like Adam on downstream tasks.
BibTeX
@inproceedings{li2022robust,
  author = {Zhiyuan Li and Srinadh Bhojanapalli and Manzil Zaheer and Sashank J. Reddi and Sanjiv Kumar},
  title = {{Robust Training of Neural Networks using Scale Invariant Architectures}},
  booktitle = {International Conference on Machine Learning},
  year = {2022},
  url = {https://openreview.net/forum?id=_LSJZP3B-b},
}
Figure