Extensible and Efficient Proxy for Neural Architecture Search

Yuhong Li, Jiajie Li, Cong Hao, Pan Li, Jinjun Xiong, Deming Chen

Introduction

Neural Architecture Search (NAS) has revolutionized the design of deep neural networks, yet its computational cost remains a significant barrier. Traditional methods often require training thousands of candidate architectures to completion, a process that can consume thousands of GPU hours. To mitigate this, researchers have turned to "proxies"—efficient shortcuts that estimate a model's performance without full training. However, existing zero-cost proxies often struggle with inconsistency across diverse search spaces and a lack of extensibility to non-classification tasks.

The Eproxy Framework

To address these limitations, this research introduces Eproxy, a "near-zero-cost" proxy designed for both efficiency and versatility. Unlike traditional proxies that rely on static architectural traits, Eproxy evaluates models through a brief, self-supervised regression task.

The Barrier Layer Innovation

A critical challenge in few-shot proxy evaluation is that simple tasks often fail to distinguish between high- and low-quality architectures. Eproxy overcomes this by introducing a barrier layer—a randomly initialized, frozen convolutional layer appended to the end of the candidate network. This layer increases the complexity of the optimization landscape, preventing poor architectures from "cheating" on simple tasks.

Comparison of loss landscapes between target training (left) and Eproxy (right). Eproxy effectively mimics the relative ranking of architectures in just 10 iterations.

As shown in the visualization above, the barrier layer transforms the loss surface into a rugged terrain. In just 10 iterations of backpropagation, superior architectures (like Arch 2) demonstrate a clear advantage in navigating this landscape compared to inferior ones (Arch 1), mirroring the results of 100,000+ iterations of standard training.

Discrete Proxy Search (DPS)

Because different downstream tasks (e.g., segmentation vs. object detection) may require different proxy configurations, the authors developed Discrete Proxy Search (DPS). DPS utilizes an evolutionary algorithm to automatically tune Eproxy's hyperparameters—such as learning rates and synthetic label types—using only a small subset of benchmarked architectures. This ensures that the proxy remains "extensible" and accurate across a wide variety of domains.

Experimental Results

The effectiveness of the Eproxy+DPS approach was validated across numerous benchmarks, including NAS-Bench-101/201, NDS, and complex multi-task spaces like NAS-Bench-MR.

Accuracy and Speed

On the NDS-ImageNet search space, Eproxy+DPS achieved a Spearman ranking correlation ($\rho$) of 0.73, a substantial improvement over the previous state-of-the-art of 0.56. More impressively, the method provides a massive leap in efficiency.

Efficiency comparison: Eproxy+DPS achieves high Spearman correlation significantly faster than traditional methods, demonstrating a 146x speedup.

As illustrated in the figure above, Eproxy+DPS reaches high correlation levels in a fraction of the time required by methods like early stopping or auto-encoder-based proxies. On the Trans-Micro benchmark, it delivered a 146x speedup while maintaining competitive performance. In practical terms, Eproxy found a superior architecture for DARTS-ImageNet in just one GPU hour using a single batch of data.

Conclusion

Eproxy and DPS represent a significant step toward making Neural Architecture Search accessible and scalable. By leveraging self-supervised tasks with intentional complexity and automated proxy tuning, this framework provides a robust, extensible, and incredibly fast alternative to traditional NAS evaluation methods. Future work aims to expand these "foundation proxies" to even more diverse modalities beyond computer vision.

Summary updated

Abstract

Efficient or near-zero-cost proxies were proposed recently to address the demanding computational issues of Neural Architecture Search (NAS) in designing deep neural networks (DNNs), where each candidate architecture network only requires one iteration of backpropagation. The values obtained from proxies are used as predictions of architecture performance for downstream tasks. However, two significant drawbacks hinder the wide adoption of these efficient proxies: (1) they are not adaptive to various NAS search spaces; and (2) they are not extensible to multi-modality downstream tasks. To address these two issues, we first propose an Extensible proxy (Eproxy) that utilizes self-supervised, few-shot training to achieve near-zero costs. A key component to our Eproxy’s efficiency is the introduction of a barrier layer with randomly initialized frozen convolution parameters, which adds non-linearities to the optimization spaces so that Eproxy can discriminate the performance of architectures at an early stage. We further propose a Discrete Proxy Search (DPS) method to find the optimized training settings for Eproxy with only a handful of benchmarked architectures on the target tasks. Our extensive experiments confirm the effectiveness of both Eproxy and DPS. On the NDS-ImageNet search spaces, Eproxy+DPS achieves a higher average ranking correlation (Spearman ρ = 0.73) than the previous efficient proxy (Spearman ρ = 0.56). On the NAS-Bench-Trans-Micro search spaces with seven tasks, Eproxy+DPS delivers comparable performance with the early stopping method (146× faster). For the end-to-end task such as DARTS-ImageNet-1k, our method delivers better results than NAS performed on CIFAR-10 while only requiring one GPU hour with a single batch of CIFAR-10 images. Our code is available at https://github.com/leeyeehoo/GenNAS-Zero.
BibTeX
@inproceedings{li2023extensible,
  author = {Yuhong Li and Jiajie Li and Cong Hao and Pan Li and Jinjun Xiong and Deming Chen},
  title = {{Extensible and Efficient Proxy for Neural Architecture Search}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.00570},
}
Figure