Contrastive Neural Architecture Search with Neural Architecture Comparators

Yaofo Chen, Yong Guo, Qi Chen, Minli Li, Yaowei Wang, Wei Zeng, Mingkui Tan

Introduction

Neural Architecture Search (NAS) aims to automate the design of optimal neural network structures. However, evaluating candidate architectures remains a significant bottleneck. Traditional methods typically rely on training a "supernet" or a regression model to predict absolute performance. These approaches suffer from high computational costs and instability due to noisy performance signals caused by random training dynamics.

Comparison between standard performance estimation and the proposed neural architecture comparator.

The authors of this paper propose a different approach: instead of predicting exact accuracy, they focus on relative comparison. By determining if one architecture is simply better than another, the search process becomes more stable and data-efficient. This leads to the development of Contrastive Neural Architecture Search (CTNAS).

Methods: Contrastive Neural Architecture Search (CTNAS)

The CTNAS framework shifts the paradigm from absolute prediction to relative comparison using three core components: the Neural Architecture Comparator (NAC), a curriculum learning strategy, and semi-supervised data exploration.

Overview of the CTNAS framework and the internal architecture of the NAC.

Neural Architecture Comparator (NAC)

The NAC is the heart of the system. Instead of outputting a single accuracy score, it takes two architecture graphs (represented as Directed Acyclic Graphs) as inputs. It employs a Graph Convolutional Network (GCN) to extract features from both, concatenates these features, and outputs the probability that the first architecture outperforms the second. This comparative approach is inherently more stable than absolute prediction.

Curriculum Learning and Data Exploration

CTNAS uses a reinforcement learning controller to propose new architectures. The "reward" for the controller is the NAC's probability that the new architecture beats a specific baseline. To keep the agent learning effectively, the baseline is periodically updated to the best architecture found so far—a curriculum learning approach that makes the search progressively harder.

To further reduce the need for expensive ground-truth data, the authors implement a data exploration technique. The NAC evaluates unlabeled architectures during the search; if it is highly confident in a comparison, that pair is pseudo-labeled and added back into the training set to improve the NAC on the fly.

Experimental Evaluation

CTNAS was evaluated across three standard search spaces: NAS-Bench-101 (CIFAR-10), a MobileNetV3-like space (ImageNet), and the DARTS cell-based space (CIFAR-10).

Key Results

  • Superior Ranking Accuracy: On NAS-Bench-101, CTNAS achieved a Kendall’s Tau (ranking correlation) of 0.751, vastly outperforming existing predictors like ReNAS (0.634) and SPOS (0.195).
  • State-of-the-Art Efficiency: In the MobileNetV3 space on ImageNet, CTNAS found an architecture achieving 77.3% Top-1 accuracy with only 0.1 GPU days of search time, requiring only 1,000 architecture queries.
  • Blazing Fast Evaluation: Because the NAC uses a lightweight 3-layer GCN, it can evaluate 100 architectures in just 4.1 milliseconds, compared to 2.7 seconds for ENAS.

Conclusion and Limitations

By leveraging relative comparisons, CTNAS provides a more stable and efficient path for neural architecture search. The approach is theoretically grounded, with proofs showing that training with binary cross-entropy on pairs is statistically consistent with optimizing global rankings.

While highly efficient, CTNAS still requires an initial set of architectures with ground-truth performance to bootstrap the NAC. Future work may extend this contrastive approach to multi-objective NAS, comparing architectures across a Pareto front of accuracy versus hardware latency.

Summary updated

BibTeX
@inproceedings{chen2021contrastive,
  author = {Yaofo Chen and Yong Guo and Qi Chen and Minli Li and Yaowei Wang and Wei Zeng and Mingkui Tan},
  title = {{Contrastive Neural Architecture Search with Neural Architecture Comparators}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  doi = {10.1109/cvpr46437.2021.00938},
}
Figure