Abstract
This paper presents a unified theoretical framework for stochastic gradient descent (SGD) variants that employ gradient masking and parameter perturbations, such as Dropout and Slimmable Networks. By establishing convergence criteria based on norm preservation and gradient alignment, the authors introduce the Alternating Training Scheme (ATS). ATS enables the simultaneous training of a full-size "super-network" and high-performing "core" sub-networks. Empirical results demonstrate that this co-training approach allows sub-networks to significantly outperform those trained in isolation, while reducing computational overhead compared to traditional multi-capacity training methods.
Introduction
Modern deep learning often utilizes heuristics to improve training efficiency or to deploy models across varying hardware capacities. Techniques such as Dropout (randomly deactivating neurons), meProp (sparse gradient updates), and Slimmable Networks (training sub-networks of varying widths) have become standard practice.
Despite their empirical success, these methods frequently lack rigorous theoretical convergence guarantees. Furthermore, training multi-capacity models typically requires multiple forward and backward passes per optimization step, leading to high computational costs. This research seeks to provide a unified theory explaining these methods and proposes a more efficient training paradigm.
Theoretical Framework: Partial SGD
The authors propose a generic template called Partial SGD, which modifies standard SGD through two primary mechanisms:
- Gradient Masking ($p_t$): Only a specific subset of parameters is updated at each iteration.
- Parameter Perturbation ($\delta x_t$): Gradients are calculated at a slightly shifted version of the current parameters.
To ensure convergence, the authors identify three critical "Criteria for Success":
- Norm Preservation (Masking): The mask must not drastically reduce the gradient's magnitude.
- Norm Preservation (Perturbation): Parameter shifts should not cause the gradient to vanish.
- Gradient Alignment: The gradient at the perturbed point must remain sufficiently aligned with the true gradient direction.
The Alternating Training Scheme (ATS)
Building on the Partial SGD theory, the researchers introduce the Alternating Training Scheme (ATS). Unlike Slimmable Networks, which calculate gradients for all sub-networks at every step, ATS alternates updates. On even steps, it updates the full "super-network" ($\mathcal{M}_{super}$); on odd steps, it updates a specific "core" sub-network ($\mathcal{M}_{core}$).
This approach is approximately twice as computationally efficient as traditional slimmable training while achieving comparable accuracy.
Key Insights and Empirical Results
The Synergy Effect
A primary finding is the "synergy" between the super-network and its sub-components. Training a small sub-network within a larger one results in performance that exceeds standalone training. For instance, on the IWSLT14 translation task, a 32x rank-reduced Transformer achieved a 24.03 BLEU score via ATS, compared to only 17.95 BLEU when trained in isolation.
Implicit Regularization and Alignment
The authors hypothesize that SGD naturally promotes alignment between sub-networks. During co-training, the gradients of the small and large models begin to point in similar directions, allowing the larger model to effectively guide the optimization of the smaller one.
Computational Efficiency
In ResNet-18 experiments on CIFAR-10, ATS matched the accuracy of Slimmable Networks while requiring significantly fewer resources: 31.3 x 10¹² FLOPs compared to 62.7 x 10¹² FLOPs for the standard approach.
Conclusion
This work provides a unified view of disparate methods like Extragradient and Dropout under the Partial SGD template. While the framework proves convergence, future research is needed to understand the "implicit bias" of these training paths on final model generalization. Additionally, the convergence criteria identified here could pave the way for dynamic architectures that switch sub-networks in real-time based on hardware constraints.
Summary updated