Distilldarts: Network Distillation for Smoothing Gradient Distributions in Differentiable Architecture Search
Lists
Only you see this
Abstract
Recent studies show that differentiable architecture search (DARTS) suffers notable instability and collapse issue: skip-connect may gradually dominate the cell, leading to deteri-orating architectures. We conjecture that the domination of skip-connect is due to its superiority in gradient compen-sate. On this foundation, we propose a novel and stable method, called DistillDARTS, to stabilize DARTS by knowl-edge distillation and self-distillation scheme. Specifically, the distillation is able to serve as a substitute for skip-connect and smooth the back-propagated gradient distributions among layers of DARTS. By compensating gradients in shallow lay-ers, our method can relieve the dependence of gradient on skip-connect and hence mitigates the collapse issue. Exten-sive experiments on a range of benchmarks demonstrate that DistillDARTS can obtain sturdy architectures with few skip-connects without additional manual interventions, thus suc-cessfully improving the robustness of DARTS. Due to the im-proved stability, our proposed approach achieves the accuracy of 97.57% on CIFAR-10 and 75.8% on ImageNet.
BibTeX
@inproceedings{liao2022distilldarts,
author = {Wenlong Liao and Zhexi Zhang and Xiaoxing Wang and Tao Yin and Huanxi Liu and Zhenyu Ren and Jian Yin and Shuzhi Feng},
title = {{Distilldarts: Network Distillation for Smoothing Gradient Distributions in Differentiable Architecture Search}},
booktitle = {ICME},
year = {2022},
url = {https://openreview.net/forum?id=6DG42-_Hi45},
}Loading related content...