TL;DR
This paper introduces RARTS, a novel differentiable architecture search (DAS) method that addresses the efficiency and accuracy limitations of existing DARTS approaches. By employing a single-level optimization framework with data and network splitting, RARTS achieves comparable or better accuracy with significantly reduced search time, and also mitigates the "architecture collapse" problem.
What Problem Does This Paper Address?
Neural Architecture Search (NAS) aims to automate the design of optimal neural network architectures. Differentiable Architecture Search (DARTS) revolutionized this field by formulating architecture search as a continuous optimization problem, significantly reducing computational cost compared to earlier methods. However, DARTS has two main variants: first-order DARTS, which is faster but less accurate and prone to convergence issues, and second-order DARTS, which is more accurate but computationally expensive due to its reliance on mixed second derivatives. Furthermore, both variants can suffer from "architecture collapse," where the search favors excessive skip-connections, degrading performance. This paper tackles these challenges by proposing a new DAS method that is both efficient and accurate, while also being more robust to architecture collapse.
Methods
The core innovation of RARTS lies in its single-level optimization framework and its "network splitting" technique. Instead of the bilevel optimization used in DARTS, RARTS formulates a relaxed Lagrangian function that combines two loss functions: one for a primary network (trained on a validation split of the data) and one for an auxiliary network (trained on a training split). These two networks share the same architecture but have different, though related, weights. This "network splitting" allows for more robust learning of architecture parameters. Additionally, RARTS utilizes the entire dataset for architecture learning by splitting it into training and validation sets, unlike first-order DARTS which uses only half the data. The method also incorporates an optional latency penalty to control model size. The optimization proceeds in an alternating fashion, updating weights and architecture parameters.
Datasets
The paper evaluates RARTS on several standard datasets:
- CIFAR-10: Used for topology search and width search experiments.
- ImageNet: Used for transfer learning experiments to evaluate the generalizability of architectures found on CIFAR-10.
- NATS-Bench: A public benchmark for architecture search, used to compare RARTS against DARTS on a standardized search space.
Key Insights and Ideas
- Network Splitting for Robustness: Using two networks with slightly different weights to learn a shared architecture provides a more stable and robust optimization landscape compared to using a single network. This helps avoid spurious equilibria and improves convergence.
- Single-Level Optimization Efficiency: Moving from a bilevel to a single-level optimization problem, combined with efficient gradient updates, significantly reduces the computational cost of the search process.
- Mitigating Architecture Collapse: By using both training and validation data for architecture updates and by employing network splitting, RARTS is shown to effectively avoid the architecture collapse problem, leading to more balanced and performant architectures.
- Unified Approach for Topology and Width Search: RARTS demonstrates its versatility by successfully applying to both searching the connectivity and operations (topology) of neural networks and searching for the optimal number of channels (width) in convolutional layers.
Key Results
- Topology Search on CIFAR-10: RARTS achieved a higher accuracy and a 60% reduction in computational cost compared to second-order DARTS. The found model was also smaller.
- Transfer Learning to ImageNet: Architectures found by RARTS performed better than those found by DARTS and its variants.
- NATS-Bench Performance: RARTS significantly outperformed DARTS-1 and DARTS-2 on NATS-Bench, achieving substantially lower error rates and, crucially, zero skip-connections in the discovered architectures.
- Width Search (Channel Pruning): On CIFAR-10 and CIFAR-100, RARTS outperformed established methods like Network Slimming and TAS, achieving better accuracy with comparable or better pruning ratios.
- Search Efficiency: RARTS consistently demonstrated higher search efficiency, requiring significantly fewer GPU days than second-order DARTS.
Limitations and Future Work
- Hyperparameter Sensitivity: While RARTS offers flexibility, the paper notes that hyperparameters like $\lambda$ and $\beta$ can impact the latency of the found models. Table 2 shows that latency weight is a particularly sensitive hyperparameter.
- Future Work: The authors plan to incorporate search space sampling and regularization techniques to further accelerate RARTS and explore its application in broader deep learning domains.
Summary updated