Abstract
State-of-the-art training algorithms for deep learning models are based on stochastic gradient descent (SGD). Recently, many variations have been explored: perturbing parameters for better accuracy (such as in Extragradient), limiting SGD updates to a subset of parameters for increased efficiency (such as meProp) or a combination of both (such as Dropout). However, the convergence of these methods is often not studied in theory. We propose a unified theoretical framework to study such SGD variants—encompassing the aforementioned algorithms and additionally a broad variety of methods used for communication efficient training or model compression. Our insights can be used as a guide to improve the efficiency of such methods and facilitate generalization to new applications. As an example, we tackle the task of jointly training networks, a version of which (limited to sub-networks) is used to create Slimmable Networks. By training a low-rank Transformer jointly with a standard one we obtain superior performance than when it is trained separately.
Introduction
In the pursuit of computational efficiency and improved generalization, researchers have developed numerous variants of Stochastic Gradient Descent (SGD). These methods often involve perturbing parameters (e.g., Extragradient) or updating only a subset of weights (e.g., meProp and Dropout). While empirically successful, these "partial gradient" methods frequently lack a rigorous theoretical foundation, making it difficult to predict their convergence behavior or adapt them to new architectures. This work, presented at AISTATS 2022, introduces a unified framework called Partial SGD to bridge this gap, explaining the mechanics behind these disparate techniques and providing a path toward more efficient training schemes.
Theoretical Framework: Partial SGD
The core contribution is the introduction of Partial SGD, a generic algorithmic template that encapsulates various training tricks through two primary mechanisms:
- Masking ($p_t$): A binary mask determining which parameters are updated at each iteration.
- Perturbation ($\delta x_t$): A temporary parameter shift used solely for gradient calculation, allowing the optimizer to probe the local weight space.
To guarantee convergence in non-convex settings, the authors identify three essential criteria: Masking Preservation, Perturbation Preservation, and Gradient Alignment. The latter ensures that the masked/perturbed gradient maintains a positive correlation with the true gradient of the full model.
Building on this framework, the authors propose the Alternating Training Scheme (ATS). ATS alternates updates between a "core" sub-network and the full "super-network," ensuring their optimization paths remain aligned during training.
Empirical Insights and Gradient Alignment
A pivotal finding of this research is the role of gradient alignment. The authors demonstrate that the success of partial gradient methods hinges on how well the sub-network's updates align with the global objective. If the gradients point in conflicting directions, training fails to converge effectively.
Interestingly, empirical analysis reveals that Dropout acts as a regularizer for alignment. As shown in Figure 6, training with Dropout maintains a significantly higher level of gradient alignment over time compared to training without it. This suggests that Dropout's efficacy stems from forcing different network components to converge on a consistent descent direction. Furthermore, the study shows that small models, such as low-rank Transformers, learn superior features when trained as sub-networks of a larger "teacher" model through shared weights.
Key Results
The framework and ATS were validated on standard benchmarks including IWSLT14/WMT17 for translation and CIFAR-10 for image classification.
- Performance Gains: On IWSLT14, a low-rank Transformer (32x rank reduction) trained via ATS achieved a BLEU score of 24.03, a substantial improvement over the 17.95 BLEU score achieved when trained in isolation.
- Efficiency: For ResNet-18 on CIFAR-10, ATS matched the accuracy of "Slimmable Networks" while requiring only half the FLOPs ($31.3 \times 10^{12}$ vs $62.7 \times 10^{12}$).
- Theoretical Consistency: The authors successfully derived convergence rates for Partial SGD that match the best-known bounds for standard SGD, provided the masking and perturbation criteria are met.
Conclusion
By unifying disparate training methods under the Partial SGD framework, this work provides both theoretical clarity and practical tools for efficient deep learning. While the current results on Transformers and ResNets are promising, future work may explore the sensitivity of hyperparameters like stepsize scaling and the applicability of ATS to specialized architectures. The potential for dynamic switching in distributed environments remains a compelling avenue for further research.
Summary updated