Distilldarts: Network Distillation for Smoothing Gradient Distributions in Differentiable Architecture Search

Wenlong Liao, Zhexi Zhang, Xiaoxing Wang, Tao Yin, Huanxi Liu, Zhenyu Ren, Jian Yin, Shuzhi FengICME 2022 · OpenReview

Abstract

Recent studies show that differentiable architecture search (DARTS) suffers notable instability and collapse issue: skip-connect may gradually dominate the cell, leading to deteri-orating architectures. We conjecture that the domination of skip-connect is due to its superiority in gradient compen-sate. On this foundation, we propose a novel and stable method, called DistillDARTS, to stabilize DARTS by knowl-edge distillation and self-distillation scheme. Specifically, the distillation is able to serve as a substitute for skip-connect and smooth the back-propagated gradient distributions among layers of DARTS. By compensating gradients in shallow lay-ers, our method can relieve the dependence of gradient on skip-connect and hence mitigates the collapse issue. Exten-sive experiments on a range of benchmarks demonstrate that DistillDARTS can obtain sturdy architectures with few skip-connects without additional manual interventions, thus suc-cessfully improving the robustness of DARTS. Due to the im-proved stability, our proposed approach achieves the accuracy of 97.57% on CIFAR-10 and 75.8% on ImageNet.
BibTeX
@inproceedings{liao2022distilldarts,
  author = {Wenlong Liao and Zhexi Zhang and Xiaoxing Wang and Tao Yin and Huanxi Liu and Zhenyu Ren and Jian Yin and Shuzhi Feng},
  title = {{Distilldarts: Network Distillation for Smoothing Gradient Distributions in Differentiable Architecture Search}},
  booktitle = {ICME},
  year = {2022},
  url = {https://openreview.net/forum?id=6DG42-_Hi45},
}

Loading related content...

Figure