Simultaneous Training of Partially Masked Neural Networks

Amirkeivan Mohtashami, Martin Jaggi, Sebastian U. Stich

Abstract

This paper presents a unified theoretical framework for stochastic gradient descent (SGD) variants that employ gradient masking and parameter perturbations, such as Dropout and Slimmable Networks. By establishing convergence criteria based on norm preservation and gradient alignment, the authors introduce the Alternating Training Scheme (ATS). ATS enables the simultaneous training of a full-size "super-network" and high-performing "core" sub-networks. Empirical results demonstrate that this co-training approach allows sub-networks to significantly outperform those trained in isolation, while reducing computational overhead compared to traditional multi-capacity training methods.

Introduction

Modern deep learning often utilizes heuristics to improve training efficiency or to deploy models across varying hardware capacities. Techniques such as Dropout (randomly deactivating neurons), meProp (sparse gradient updates), and Slimmable Networks (training sub-networks of varying widths) have become standard practice.

Despite their empirical success, these methods frequently lack rigorous theoretical convergence guarantees. Furthermore, training multi-capacity models typically requires multiple forward and backward passes per optimization step, leading to high computational costs. This research seeks to provide a unified theory explaining these methods and proposes a more efficient training paradigm.

Theoretical Framework: Partial SGD

The authors propose a generic template called Partial SGD, which modifies standard SGD through two primary mechanisms:

  1. Gradient Masking ($p_t$): Only a specific subset of parameters is updated at each iteration.
  2. Parameter Perturbation ($\delta x_t$): Gradients are calculated at a slightly shifted version of the current parameters.

To ensure convergence, the authors identify three critical "Criteria for Success":

  • Norm Preservation (Masking): The mask must not drastically reduce the gradient's magnitude.
  • Norm Preservation (Perturbation): Parameter shifts should not cause the gradient to vanish.
  • Gradient Alignment: The gradient at the perturbed point must remain sufficiently aligned with the true gradient direction.

The Alternating Training Scheme (ATS)

Building on the Partial SGD theory, the researchers introduce the Alternating Training Scheme (ATS). Unlike Slimmable Networks, which calculate gradients for all sub-networks at every step, ATS alternates updates. On even steps, it updates the full "super-network" ($\mathcal{M}_{super}$); on odd steps, it updates a specific "core" sub-network ($\mathcal{M}_{core}$).

Figure 3: Visualization of the relationship between the super-network and core sub-networks across different layers.

This approach is approximately twice as computationally efficient as traditional slimmable training while achieving comparable accuracy.

Key Insights and Empirical Results

The Synergy Effect

A primary finding is the "synergy" between the super-network and its sub-components. Training a small sub-network within a larger one results in performance that exceeds standalone training. For instance, on the IWSLT14 translation task, a 32x rank-reduced Transformer achieved a 24.03 BLEU score via ATS, compared to only 17.95 BLEU when trained in isolation.

Implicit Regularization and Alignment

The authors hypothesize that SGD naturally promotes alignment between sub-networks. During co-training, the gradients of the small and large models begin to point in similar directions, allowing the larger model to effectively guide the optimization of the smaller one.

Figure 6: Empirical evidence of gradient alignment over training epochs, comparing models with and without Dropout.

Computational Efficiency

In ResNet-18 experiments on CIFAR-10, ATS matched the accuracy of Slimmable Networks while requiring significantly fewer resources: 31.3 x 10¹² FLOPs compared to 62.7 x 10¹² FLOPs for the standard approach.

Conclusion

This work provides a unified view of disparate methods like Extragradient and Dropout under the Partial SGD template. While the framework proves convergence, future research is needed to understand the "implicit bias" of these training paths on final model generalization. Additionally, the convergence criteria identified here could pave the way for dynamic architectures that switch sub-networks in real-time based on hardware constraints.

Summary updated

BibTeX
@inproceedings{mohtashami2022simultaneous,
  author = {Amirkeivan Mohtashami and Martin Jaggi and Sebastian U. Stich},
  title = {{Simultaneous Training of Partially Masked Neural Networks}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2022},
  url = {https://openreview.net/forum?id=YmqDMdxpPUT},
}
Figure