AlphaNet: Improved Training of Supernets with Alpha-Divergence

Dilin Wang, Chengyue Gong, Meng Li, Qiang Liu, Vikas Chandra

TL;DR

AlphaNet introduces a more robust way to train "supernets"—large neural networks that contain millions of smaller sub-networks—by replacing standard KL divergence with a more flexible family of measures called $\alpha$-divergence. By adaptively penalizing both the over-estimation and under-estimation of a teacher model's uncertainty, AlphaNet produces sub-networks that set new state-of-the-art performance benchmarks for efficient computer vision.

The Challenge of Supernet Training

In the world of Neural Architecture Search (NAS), training every possible candidate model from scratch is prohibitively expensive. To solve this, researchers use supernets: one massive "mother" network that shares its weights with all possible "daughter" sub-networks.

In-place Knowledge Distillation in Supernets

To make these sub-networks perform well, researchers use in-place Knowledge Distillation (KD), where the largest sub-network (the teacher) supervises the smaller ones (the students). Traditionally, this is done using KL divergence. However, KL divergence has a specific bias: it is "zero-avoiding." It heavily penalizes a student for missing a mode the teacher found (under-estimation), but it is relatively lenient when a student predicts high probability for a class the teacher thinks is impossible (over-estimation). This mismatch leads to sub-optimal sub-networks that fail to accurately capture the teacher's knowledge.

Methods: Introducing Adaptive-KD

The authors propose Adaptive-KD, which swaps KL divergence for a generalized $\alpha$-divergence to correct these biases.

  1. Adaptive Selection: Instead of a single value, the method uses the maximum of two different $\alpha$-divergences—one with a positive $\alpha$ to penalize under-estimation and one with a negative $\alpha$ to penalize over-estimation. This forces the student to "hug" the teacher's distribution more tightly from both sides.
  2. Stabilizing Gradients: Directly optimizing $\alpha$-divergence can be unstable because the gradients can explode. The authors introduced a gradient clipping technique. Crucially, they provide a mathematical proof showing that this clipped gradient still corresponds to a valid "f-divergence," ensuring the optimization remains theoretically sound.
  3. Training Routine: They follow the "sandwich rule," training the largest sub-network, the smallest sub-network, and several random ones in each iteration to ensure the entire supernet remains functional.

Key Insights and Ideas

The KL Asymmetry: The most important insight is identifying that KL divergence is a "one-way street." By visualizing how different $\alpha$ values react to student errors, the authors showed that standard KD is fundamentally biased toward over-estimating uncertainty.

Behavior of different alpha-divergences under over-estimation and under-estimation scenarios

Principled Heuristics: Often, "clipping" in deep learning is a dirty hack. Here, the authors showed that clipping the density ratio in the gradient is equivalent to minimizing a specific, convex $f$-divergence, turning a trick into a principled method.

Efficiency Gains: The method doesn't just improve accuracy; it improves the efficiency of the search. By making the supernet training more robust, the sub-networks inherited better weights, reducing the need for expensive fine-tuning.

Key Results

  • New SOTA: The "AlphaNet" family outperforms prior heavyweights like BigNAS, Once-for-All (OFA), and AttentiveNAS.
  • Efficiency Milestone: AlphaNet-A4 achieved 80.0% top-1 accuracy on ImageNet with only 444M FLOPs. For comparison, it achieves the same accuracy as the popular OFA model but uses 35% fewer FLOPs.
  • Small Model Dominance: At the very low-power end (200M FLOPs), AlphaNet-A0 achieves 77.8% accuracy, a significant jump over MobileNetV3 (75.2%) and BigNAS (76.5%).
  • Better Transfer: Models trained with AlphaNet's Adaptive-KD transferred better to specialized datasets (like Oxford Flowers, Oxford-IIIT Pets, and Stanford Cars) than standard EfficientNet models of similar size.

Limitations and Future Work

  • Hyperparameter Tuning: While the authors found their default settings ($\alpha=1, \beta=5$) worked well across tasks, the method introduces three new hyperparameters ($\alpha_+, \alpha_-, \beta$) that might require tuning for non-vision tasks.
  • Output-Only Distillation: The current work focuses on distilling the final output probabilities. The authors note that it is "less clear" how to apply this adaptive $\alpha$-divergence to intermediate feature layers, as the latent spaces of different sub-networks may not align easily.
  • Search Space Constraints: The improvements were demonstrated on the AttentiveNAS search space. Whether these gains hold for radically different architectures (like pure Transformers) remains an open question for future research.

Summary updated

Abstract

Weight-sharing neural architecture search (NAS) is an effective technique for automating efficient neural architecture design. Weight-sharing NAS builds a supernet that assembles all the architectu...
BibTeX
@inproceedings{wang2021alphanet,
  author = {Dilin Wang and Chengyue Gong and Meng Li and Qiang Liu and Vikas Chandra},
  title = {{AlphaNet: Improved Training of Supernets with Alpha-Divergence}},
  booktitle = {International Conference on Machine Learning},
  year = {2021},
  url = {https://openreview.net/forum?id=S4sQEl1k8O0},
}
Figure