TL;DR
This paper introduces RARTS, a novel single-level optimization method for neural architecture search that improves upon DARTS by addressing its efficiency and convergence issues. RARTS achieves higher accuracy and significantly faster search times by employing data and network splitting, leading to more robust and efficient architecture discovery for both network topology and width.
What Problem Does This Paper Address?
Neural Architecture Search (NAS) aims to automate the design of optimal neural network architectures. Differentiable Architecture Search (DARTS) revolutionized this field by relaxing the discrete search space into a continuous one, allowing for gradient-based optimization. However, DARTS faces challenges: first-order DARTS can struggle with convergence and accuracy, while second-order DARTS, though more accurate, is computationally expensive due to its reliance on mixed second derivatives. Furthermore, both variants can suffer from "architecture collapse," where architectures are dominated by skip-connections, degrading performance. This paper tackles these limitations by proposing RARTS, a method that aims for the efficiency of first-order methods while achieving the accuracy and robustness of second-order methods, all without modifying the search space itself.
Methods
RARTS introduces a "relaxed" single-level optimization framework. Instead of the bilevel optimization in DARTS, RARTS uses a single objective function. The core innovations are:
- Data Splitting: Similar to DARTS, RARTS splits the dataset into training ($D_t$) and validation ($D_v$) sets to prevent overfitting.
- Network Splitting: This is a key novelty. RARTS maintains two networks with different but related weights: a "primary" network $(y, \alpha)$ trained on $D_v$ and an "auxiliary" network $(w, \alpha)$ trained on $D_t$. Both networks share the same architecture parameters $\alpha$. This allows the networks to cooperate and learn from different data splits simultaneously.
- Relaxed Lagrangian: The objective function is a relaxed Lagrangian that combines the losses from both networks and includes a penalty term that encourages their weights $(y, w)$ to be close. This penalty term is crucial for enabling information exchange between the two networks.
- Latency Penalty (Optional): For hardware-aware search, RARTS can incorporate a penalty term based on the estimated latency of the architecture, allowing for the discovery of efficient models.
- Alternating Optimization: RARTS updates the weights $(w, y)$ and architecture parameters $(\alpha)$ in an alternating fashion, similar to a Gauss-Seidel method. This three-step update process is designed to be more stable and efficient than DARTS.
Datasets
The paper evaluates RARTS on several standard datasets and benchmarks:
- CIFAR-10: Used for searching network topology and for channel pruning experiments.
- CIFAR-100: Used for channel pruning experiments.
- ImageNet: Used for evaluating the transferability of architectures found on CIFAR-10.
- NATS-Bench: A benchmark for neural architecture search that provides pre-evaluated architectures, allowing for fair comparisons without local evaluation.
Key Insights and Ideas
- Two Heads are Better Than One: The core idea of using two cooperating networks (primary and auxiliary) with shared architecture parameters is a powerful way to leverage both training and validation data more effectively within a single optimization loop.
- Bridging the Gap: RARTS successfully bridges the gap between first-order and second-order DARTS by offering comparable accuracy to second-order methods with significantly improved efficiency, similar to first-order methods.
- Combating Architecture Collapse: The paper suggests that RARTS's approach, by involving both networks in weight updates and architecture learning, helps mitigate the architecture collapse problem, leading to more diverse and performant architectures.
- Algorithmic Purity: RARTS achieves its improvements purely through algorithmic advancements in the optimization process, without requiring modifications to the search space itself, making it broadly applicable.
Key Results
- Topology Search (CIFAR-10): RARTS achieved a test error of 2.65%, outperforming first-order DARTS (3.00%) and second-order DARTS (2.76%). Crucially, it did so with only 1.1 GPU days of search, a 60% reduction compared to second-order DARTS. The found model also had a smaller parameter count (3.2M vs. 3.3M).
- Transfer Learning (ImageNet): Architectures found by RARTS on CIFAR-10 transferred well to ImageNet, achieving competitive performance.
- NATS-Bench: RARTS significantly outperformed DARTS on NATS-Bench, achieving much lower error rates (e.g., 11.48% vs. 40.16% on CIFAR-10) and completely avoiding architecture collapse (0% skip-connections vs. 100% for DARTS).
- Channel Pruning (ResNet-164 on CIFAR-10/100): RARTS outperformed existing differentiable pruning methods like Network Slimming and TAS, achieving better accuracy with comparable or higher pruning ratios.
- Convergence: Analytical and experimental results on a solvable model demonstrated that RARTS converges to the true global minimum, unlike first-order DARTS which can get stuck in spurious equilibria.
Limitations and Future Work
- Hyperparameter Sensitivity: While RARTS is generally robust, the paper notes that hyperparameters like latency weight can significantly impact model latency, suggesting careful tuning is needed for hardware-aware search.
- Future Work: The authors plan to incorporate search space sampling and regularization techniques to further accelerate RARTS, making it applicable to a broader range of deep learning tasks. They also aim to explore its application in more complex search spaces.
Summary updated