RARTS: a Relaxed Architecture Search Method

Fanghui Xue, Yingyong Qi, Jack Xin

TL;DR

This paper introduces RARTS, a novel differentiable architecture search (DAS) method that addresses the efficiency and accuracy limitations of existing DARTS approaches. By employing a single-level optimization framework with data and network splitting, RARTS achieves comparable or better accuracy with significantly reduced search time, and also mitigates the "architecture collapse" problem.

What Problem Does This Paper Address?

Neural Architecture Search (NAS) aims to automate the design of optimal neural network architectures. Differentiable Architecture Search (DARTS) revolutionized this field by formulating architecture search as a continuous optimization problem, significantly reducing computational cost compared to earlier methods. However, DARTS has two main variants: first-order DARTS, which is faster but less accurate and prone to convergence issues, and second-order DARTS, which is more accurate but computationally expensive due to its reliance on mixed second derivatives. Furthermore, both variants can suffer from "architecture collapse," where the search favors excessive skip-connections, degrading performance. This paper tackles these challenges by proposing a new DAS method that is both efficient and accurate, while also being more robust to architecture collapse.

Methods

The core innovation of RARTS lies in its single-level optimization framework and its "network splitting" technique. Instead of the bilevel optimization used in DARTS, RARTS formulates a relaxed Lagrangian function that combines two loss functions: one for a primary network (trained on a validation split of the data) and one for an auxiliary network (trained on a training split). These two networks share the same architecture but have different, though related, weights. This "network splitting" allows for more robust learning of architecture parameters. Additionally, RARTS utilizes the entire dataset for architecture learning by splitting it into training and validation sets, unlike first-order DARTS which uses only half the data. The method also incorporates an optional latency penalty to control model size. The optimization proceeds in an alternating fashion, updating weights and architecture parameters.

Datasets

The paper evaluates RARTS on several standard datasets:

  • CIFAR-10: Used for topology search and width search experiments.
  • ImageNet: Used for transfer learning experiments to evaluate the generalizability of architectures found on CIFAR-10.
  • NATS-Bench: A public benchmark for architecture search, used to compare RARTS against DARTS on a standardized search space.

Key Insights and Ideas

  • Network Splitting for Robustness: Using two networks with slightly different weights to learn a shared architecture provides a more stable and robust optimization landscape compared to using a single network. This helps avoid spurious equilibria and improves convergence.
  • Single-Level Optimization Efficiency: Moving from a bilevel to a single-level optimization problem, combined with efficient gradient updates, significantly reduces the computational cost of the search process.
  • Mitigating Architecture Collapse: By using both training and validation data for architecture updates and by employing network splitting, RARTS is shown to effectively avoid the architecture collapse problem, leading to more balanced and performant architectures.
  • Unified Approach for Topology and Width Search: RARTS demonstrates its versatility by successfully applying to both searching the connectivity and operations (topology) of neural networks and searching for the optimal number of channels (width) in convolutional layers.

Key Results

  • Topology Search on CIFAR-10: RARTS achieved a higher accuracy and a 60% reduction in computational cost compared to second-order DARTS. The found model was also smaller.
  • Transfer Learning to ImageNet: Architectures found by RARTS performed better than those found by DARTS and its variants.
  • NATS-Bench Performance: RARTS significantly outperformed DARTS-1 and DARTS-2 on NATS-Bench, achieving substantially lower error rates and, crucially, zero skip-connections in the discovered architectures.
  • Width Search (Channel Pruning): On CIFAR-10 and CIFAR-100, RARTS outperformed established methods like Network Slimming and TAS, achieving better accuracy with comparable or better pruning ratios.
  • Search Efficiency: RARTS consistently demonstrated higher search efficiency, requiring significantly fewer GPU days than second-order DARTS.

Limitations and Future Work

  • Hyperparameter Sensitivity: While RARTS offers flexibility, the paper notes that hyperparameters like $\lambda$ and $\beta$ can impact the latency of the found models. Table 2 shows that latency weight is a particularly sensitive hyperparameter.
  • Future Work: The authors plan to incorporate search space sampling and regularization techniques to further accelerate RARTS and explore its application in broader deep learning domains.

Summary updated

Abstract

Differentiable architecture search (DARTS) is an effective method for data-driven neural network design based on solving a bilevel optimization problem. Despite its success in many architecture search tasks, there are still some concerns about the accuracy of first-order DARTS and the efficiency of the second-order DARTS. In this paper, we formulate a single level alternative and a relaxed architecture search (RARTS) method that utilizes the whole dataset in architecture learning via both data and network splitting, without involving mixed second derivatives of the corresponding loss functions like DARTS. In our formulation of network splitting, two networks with different but related weights cooperate in search of a shared architecture. The advantage of RARTS over DARTS is justified by a convergence theorem and an analytically solvable model. Moreover, RARTS outperforms DARTS and its variants in accuracy and search efficiency, as shown in adequate experimental results. For the task of searching topological architecture, i.e., the edges and the operations, RARTS obtains a higher accuracy and 60\% reduction of computational cost than second-order DARTS on CIFAR-10. RARTS continues to out-perform DARTS upon transfer to ImageNet and is on par with recent variants of DARTS even though our innovation is purely on the training algorithm without modifying search space. For the task of searching width, i.e., the number of channels in convolutional layers, RARTS also outperforms the traditional network pruning benchmarks. Further experiments on the public architecture search benchmark like NATS-Bench also support the preeminence of RARTS.
BibTeX
@inproceedings{xue2022rarts,
  author = {Fanghui Xue and Yingyong Qi and Jack Xin},
  title = {{RARTS: a Relaxed Architecture Search Method}},
  booktitle = {IEEE Access},
  year = {2022},
  url = {https://openreview.net/forum?id=K1N4rcpSbc},
}
Figure