Direct Neural Architecture Search on Target Tasks

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 232 papers44 conceptsfrozen 2026-10-04

Neural Architecture Search (NAS) was originally conceived as a way to automate the design of deep learning models, but for years, it suffered from a fundamental scalability problem. Because searching through billions of architectural combinations is computationally expensive, researchers relied on proxy tasks: they would search for a small "cell" on a tiny dataset (like CIFAR-10) and then stack that cell repeatedly to build a larger model for a target task (like ImageNet).

However, as Yibo Yang and Shan You point out in ISTA-NAS, this creates a "search-evaluation gap." An architecture that performs best on a small proxy often fails to maintain its superiority when scaled up to the full-sized target task. The emerging consensus across recent research is to move toward proxyless search—optimizing the architecture directly on the final dataset and at the final model depth.

The Problem with Proxies

The reliance on proxies stems from memory constraints. In differentiable NAS (like DARTS), a "supernet" containing all possible operations must be loaded into GPU memory. For a deep network on high-resolution data, this supernet is far too large for standard hardware.

The danger of the proxy approach is that it assumes the optimal local structure (the cell) is independent of the global scale and the specific data distribution. Xiong Zhang and Hongmin Xu demonstrate in DCNAS that for complex tasks like semantic segmentation, the spatial requirements of the target dataset are so unique that transferring from a classification proxy leads to significant performance loss. By searching directly on the target resolution, the model can tailor its receptive fields to the actual objects it needs to segment.

Achieving Memory Efficiency

To search directly on large-scale tasks, researchers have developed mechanisms to shrink the supernet's memory footprint without losing the ability to gradient-optimize the architecture.

  • Partial Channel Connections: Yuhui Xu and Lingxi Xie introduced PC-DARTS, which samples only a small fraction of channels (e.g., 1/4) during the search phase. This reduces memory usage by a similar factor, allowing the search to run directly on ImageNet.
  • Coordinate Descent: Instead of updating the entire supernet at once, HyeongGwon Hong and Junmo Kim proposed DARTS-CD Differentiable Architecture Search Based on Coordinate Descent, which updates only one edge of the network at a time. This reduces search costs by over 90%, enabling search on much deeper networks.
  • Kernel Reusing and Masking: In dense prediction tasks like image denoising, Marcin Mozejko and colleagues Superkernel Neural Architecture Search for Image Denoising use "superkernels"—applying learned masks to a single maximal convolution. This allows the search to optimize kernel sizes and filter counts with the memory footprint of just one convolution.

Optimizing Global Structure

Direct search also allows researchers to move beyond the "repeated cell" paradigm. In FP-NAS, Xiaoliang Dai and Zhicheng Yan argue that searching the macro-architecture—the specific configuration of every single layer—is superior to stacking cells. This global optimization ensures that resource allocation (like channel counts) is balanced across the entire depth of the network.

Similarly, Vasco Lopes and Fabio Maria Carlucci framed this as a coordination game in MANAS, where independent agents on a graph optimize different parts of the network. By searching on the final target depth, they ensure the operations are optimized for the actual scale of the model, rather than a shallow proxy.

Task-Specific Specialization

The shift toward direct search is particularly vital for downstream tasks like object detection and medical imaging. Xiaoxing Wang and Junchi Yan demonstrated with EAutoDet that searching directly on detection datasets like COCO allows the model to bypass ImageNet pre-training entirely. The resulting architectures learn features specifically tailored to spatial and semantic requirements that classification-based proxies simply cannot capture.

In specialized domains like medical imaging, Khoa Vo and Ngan Le show that proxyless search is essential for volumetric segmentation Meta-Learning of NAS for Few-shot Learning in Medical Image Applications. Direct optimization on high-resolution 3D data ensures the discovered network captures the fine-grained structural nuances necessary for clinical applications, which would be lost if the search were performed on simplified 2D proxies.

Efficiency through Bottlenecks

Finally, researchers have found that direct search doesn't have to be slow. Zhaohui Yang and Yunhe Wang introduced HourNAS, which identifies a "vital few" set of blocks that act as information bottlenecks. By prioritizing the search on these critical components while efficiently filling in the rest of the network, they achieved state-of-the-art results on ImageNet in just 3 hours of GPU time—proving that direct, proxyless search can be even faster than the proxy-based methods it replaces.

Papers in this version

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 15 papers
  2. Version 1 · 2026-10-04 · 32 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_direct_large_scale_architecture_search_da55c675,
  title = {direct large-scale architecture search},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/direct-large-scale-architecture-search/rdr_da55c675398f4f068fb2344a4ebe0596},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_direct_neural_architecture_search_on_tar_da55c675_v1,
  title = {Direct Neural Architecture Search on Target Tasks},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/direct-neural-architecture-search-on-target-tasks/rdr_da55c675398f4f068fb2344a4ebe0596/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}