Introduction
Neural Architecture Search (NAS) has revolutionized the design of deep neural networks, yet its computational cost remains a significant barrier. Traditional methods often require training thousands of candidate architectures to completion, a process that can consume thousands of GPU hours. To mitigate this, researchers have turned to "proxies"—efficient shortcuts that estimate a model's performance without full training. However, existing zero-cost proxies often struggle with inconsistency across diverse search spaces and a lack of extensibility to non-classification tasks.
The Eproxy Framework
To address these limitations, this research introduces Eproxy, a "near-zero-cost" proxy designed for both efficiency and versatility. Unlike traditional proxies that rely on static architectural traits, Eproxy evaluates models through a brief, self-supervised regression task.
The Barrier Layer Innovation
A critical challenge in few-shot proxy evaluation is that simple tasks often fail to distinguish between high- and low-quality architectures. Eproxy overcomes this by introducing a barrier layer—a randomly initialized, frozen convolutional layer appended to the end of the candidate network. This layer increases the complexity of the optimization landscape, preventing poor architectures from "cheating" on simple tasks.
As shown in the visualization above, the barrier layer transforms the loss surface into a rugged terrain. In just 10 iterations of backpropagation, superior architectures (like Arch 2) demonstrate a clear advantage in navigating this landscape compared to inferior ones (Arch 1), mirroring the results of 100,000+ iterations of standard training.
Discrete Proxy Search (DPS)
Because different downstream tasks (e.g., segmentation vs. object detection) may require different proxy configurations, the authors developed Discrete Proxy Search (DPS). DPS utilizes an evolutionary algorithm to automatically tune Eproxy's hyperparameters—such as learning rates and synthetic label types—using only a small subset of benchmarked architectures. This ensures that the proxy remains "extensible" and accurate across a wide variety of domains.
Experimental Results
The effectiveness of the Eproxy+DPS approach was validated across numerous benchmarks, including NAS-Bench-101/201, NDS, and complex multi-task spaces like NAS-Bench-MR.
Accuracy and Speed
On the NDS-ImageNet search space, Eproxy+DPS achieved a Spearman ranking correlation ($\rho$) of 0.73, a substantial improvement over the previous state-of-the-art of 0.56. More impressively, the method provides a massive leap in efficiency.
As illustrated in the figure above, Eproxy+DPS reaches high correlation levels in a fraction of the time required by methods like early stopping or auto-encoder-based proxies. On the Trans-Micro benchmark, it delivered a 146x speedup while maintaining competitive performance. In practical terms, Eproxy found a superior architecture for DARTS-ImageNet in just one GPU hour using a single batch of data.
Conclusion
Eproxy and DPS represent a significant step toward making Neural Architecture Search accessible and scalable. By leveraging self-supervised tasks with intentional complexity and automated proxy tuning, this framework provides a robust, extensible, and incredibly fast alternative to traditional NAS evaluation methods. Future work aims to expand these "foundation proxies" to even more diverse modalities beyond computer vision.
Summary updated