Abstract
Differentiable Architecture Search (DARTS) has revolutionized Neural Architecture Search (NAS) by enabling efficient gradient-based optimization. However, DARTS frequently suffers from "model collapse," where the search process unfairly favors parameter-free operations like skip-connections. This paper introduces EC-DARTS, which addresses the "operation gap" and "structure gap" through two novel mechanisms: Cross-Edge Normalization (CEN) and Induced Structural Transition (IST). Our approach significantly improves the correlation between search and retraining performance, achieving state-of-the-art results on CIFAR and ImageNet benchmarks with minimal computational cost.
Introduction
Neural Architecture Search (NAS) aims to automate the design of high-performance neural networks. DARTS, a popular differentiable NAS method, relaxes the discrete search space into a continuous one, allowing for efficient optimization. Despite its efficiency, DARTS often fails to find optimal architectures due to a bias toward simple operations, a phenomenon known as model collapse. This bias stems from two fundamental issues: the operation gap and the structure gap.
Addressing the Gaps in DARTS
The Operation Gap arises because different operations compete on the same edge. Parameter-free operations (e.g., skip-connections) converge faster than parameter-heavy ones (e.g., convolutions), leading the optimizer to prematurely favor the former. The Structure Gap refers to the discrepancy between the dense "supernet" used during search and the sparse, pruned architecture used during final retraining. An architecture that performs well in a multi-path supernet may not be optimal when isolated.
Proposed Methodology: CEN and IST
To mitigate these gaps, EC-DARTS introduces two key components:
- Cross-Edge Normalization (CEN): Instead of local competition on a single edge, CEN groups operations by category across all edges. By normalizing weights within these categories, the search focuses on the relative importance of an operation type across the entire network, ensuring a fairer competition between convolutions and identity mappings.
- Induced Structural Transition (IST): IST bridges the structure gap by introducing an auxiliary sparse model during the search phase. This auxiliary model, containing only the top-k operations, is trained alongside the supernet. This "induces" the search process to favor structures that maintain high performance in a sparse configuration.
Experimental Evaluation
We evaluated EC-DARTS on CIFAR-10, CIFAR-100, and ImageNet. A critical metric for NAS reliability is the Kendall Tau correlation, which measures the rank correlation between search-phase performance and final retraining accuracy.
As shown in the experimental results, EC-DARTS dramatically improves this correlation. While standard DARTS variants often show near-zero or even negative correlation, EC-DARTS achieves a Kendall Tau of up to 0.67, indicating a much more reliable search process.
On CIFAR-10, EC-DARTS achieved a test error of 2.39% with a search cost of only 0.3 GPU days. On ImageNet, it reached a top-1 error of 23.6% in the mobile setting, outperforming both manual designs and other NAS methods.
Conclusion
EC-DARTS effectively addresses the inherent biases in differentiable NAS by equalizing operation competition and aligning the search and retraining structures. The introduction of CEN and IST leads to a more stable and consistent search process, as evidenced by the significantly improved Kendall Tau scores and state-of-the-art performance across multiple benchmarks.
Summary updated