EC-DARTS: Inducing Equalized and Consistent Optimization into DARTS

Qinqin Zhou, Xiawu Zheng, Liujuan Cao, Bineng Zhong, Teng Xi, Gang Zhang, Errui Ding, Mingliang Xu, Rongrong Ji

Abstract

Differentiable Architecture Search (DARTS) has revolutionized Neural Architecture Search (NAS) by enabling efficient gradient-based optimization. However, DARTS frequently suffers from "model collapse," where the search process unfairly favors parameter-free operations like skip-connections. This paper introduces EC-DARTS, which addresses the "operation gap" and "structure gap" through two novel mechanisms: Cross-Edge Normalization (CEN) and Induced Structural Transition (IST). Our approach significantly improves the correlation between search and retraining performance, achieving state-of-the-art results on CIFAR and ImageNet benchmarks with minimal computational cost.

Introduction

Neural Architecture Search (NAS) aims to automate the design of high-performance neural networks. DARTS, a popular differentiable NAS method, relaxes the discrete search space into a continuous one, allowing for efficient optimization. Despite its efficiency, DARTS often fails to find optimal architectures due to a bias toward simple operations, a phenomenon known as model collapse. This bias stems from two fundamental issues: the operation gap and the structure gap.

Addressing the Gaps in DARTS

The Operation Gap arises because different operations compete on the same edge. Parameter-free operations (e.g., skip-connections) converge faster than parameter-heavy ones (e.g., convolutions), leading the optimizer to prematurely favor the former. The Structure Gap refers to the discrepancy between the dense "supernet" used during search and the sparse, pruned architecture used during final retraining. An architecture that performs well in a multi-path supernet may not be optimal when isolated.

Proposed Methodology: CEN and IST

To mitigate these gaps, EC-DARTS introduces two key components:

  1. Cross-Edge Normalization (CEN): Instead of local competition on a single edge, CEN groups operations by category across all edges. By normalizing weights within these categories, the search focuses on the relative importance of an operation type across the entire network, ensuring a fairer competition between convolutions and identity mappings.
  2. Induced Structural Transition (IST): IST bridges the structure gap by introducing an auxiliary sparse model during the search phase. This auxiliary model, containing only the top-k operations, is trained alongside the supernet. This "induces" the search process to favor structures that maintain high performance in a sparse configuration.
Conceptual overview of Cross-Edge Normalization (CEN) and Induced Structural Transition (IST).

Experimental Evaluation

We evaluated EC-DARTS on CIFAR-10, CIFAR-100, and ImageNet. A critical metric for NAS reliability is the Kendall Tau correlation, which measures the rank correlation between search-phase performance and final retraining accuracy.

As shown in the experimental results, EC-DARTS dramatically improves this correlation. While standard DARTS variants often show near-zero or even negative correlation, EC-DARTS achieves a Kendall Tau of up to 0.67, indicating a much more reliable search process.

Comparison of Kendall Tau correlation between DARTS (2nd Order) and EC-DARTS, showing a significant increase in search reliability.

On CIFAR-10, EC-DARTS achieved a test error of 2.39% with a search cost of only 0.3 GPU days. On ImageNet, it reached a top-1 error of 23.6% in the mobile setting, outperforming both manual designs and other NAS methods.

Conclusion

EC-DARTS effectively addresses the inherent biases in differentiable NAS by equalizing operation competition and aligning the search and retraining structures. The introduction of CEN and IST leads to a more stable and consistent search process, as evidenced by the significantly improved Kendall Tau scores and state-of-the-art performance across multiple benchmarks.

Summary updated

Abstract

Based on the relaxed search space, differential architecture search (DARTS) is efficient in searching for a high-performance architecture. However, the unbalanced competition among operations that have different trainable parameters causes the model collapse. Besides, the inconsistent structures in the search and retraining stages causes cross-stage evaluation to be unstable. In this paper, we call these issues as an operation gap and a structure gap in DARTS. To shrink these gaps, we propose to induce equalized and consistent optimization in differentiable architecture search (EC-DARTS). EC-DARTS decouples different operations based on their categories to optimize the operation weights so that the operation gap between them is shrinked. Besides, we introduce an induced structural transition to bridge the structure gap between the model structures in the search and retraining stages. Extensive experiments on CIFAR10 and ImageNet demonstrate the effectiveness of our method. Specifically, on CIFAR10, we achieve a test error of 2.39%, while only 0.3 GPU days on NVIDIA TITAN V. On ImageNet, our method achieves a top-1 error of 23.6% under the mobile setting.
BibTeX
@inproceedings{zhou2021ec,
  author = {Qinqin Zhou and Xiawu Zheng and Liujuan Cao and Bineng Zhong and Teng Xi and Gang Zhang and Errui Ding and Mingliang Xu and Rongrong Ji},
  title = {{EC-DARTS: Inducing Equalized and Consistent Optimization into DARTS}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  doi = {10.1109/iccv48922.2021.01177},
}
Figure