Direct Neural Architecture Search on Target Tasks

18 concepts15 papers1 proposal

Early differentiable Neural Architecture Search (NAS) frameworks promised to automate network design by transforming a discrete search space into a continuous, differentiable one. However, these methods suffered from a fundamental bottleneck: memory explosion. To find an optimal architecture, a massive "super-network" containing all possible candidate operations (e.g., various convolutions, poolings, and skip connections) had to be loaded into GPU memory simultaneously.

Because a full-scale super-network cannot fit onto standard hardware, early practitioners relied on proxies. Instead of searching directly on the final task, they searched on smaller proxy datasets (like CIFAR-10 instead of ImageNet), shallower networks (searching a 3-layer cell and stacking it 20 times for evaluation), or simpler proxy tasks (like image classification instead of semantic segmentation).

This proxy-based approach introduces a severe proxy gap (or search-evaluation gap). An architectural cell that excels in a shallow network on a small dataset often performs poorly when scaled up to deep networks or deployed on high-resolution, complex downstream tasks. The statistics, spatial resolutions, and structural requirements of the proxy simply do not match the target environment.

To solve this, a prominent direction in NAS focuses on proxyless or direct large-scale architecture search. By drastically reducing the memory and computational footprint of the search phase, these methods allow optimization to happen directly on the target dataset, target network depth, and target task.


Shrinking the Super-Network Memory Footprint

The core technical challenge of proxyless NAS is fitting the super-network into GPU memory. Researchers have developed several distinct mathematical and architectural mechanisms to bypass the memory wall of continuous relaxation.

Path Binarization and Sampling

Instead of keeping all candidate paths active during the forward and backward passes, we can sample a subset of the network. Cai et al. introduced path binarization, which stores the architecture parameters in memory but only activates a single path during any given training step. This bounds the memory consumption of the search phase to that of a standard, single network.

Taking a structural approach, Xu et al. proposed partial channel connections. Instead of routing all channels through the expensive candidate operations, they sample a small fraction (e.g., 1/4) of the channels for the architectural search while bypassing the remaining channels directly. This reduces the memory footprint proportionally, enabling direct optimization on large-scale datasets like ImageNet.

Standard Differentiable NAS (All paths & channels active):
[Input] ---> [ Conv 3x3 ] ---> [Output]
        ---> [ Conv 5x5 ] ---> 
        ---> [ MaxPool  ] ---> 

Partial Channel Connections (Only 1/4 channels active in search):
[Input] |--(1/4 channels)--> [ Active Search Space ] --|---> [Output]
        |--(3/4 channels)----------------------------|

Algorithmic Efficiency

Other methods alter the optimization schedule itself. Hong et al. applied coordinate descent to differentiable NAS. Instead of updating the entire super-network simultaneously, their framework updates only one edge of the network at a time, slashing memory requirements and allowing direct optimization on deep target networks.

For dense prediction tasks like image denoising, where high-resolution data compounds memory issues, Mozejko et al. designed a "superkernel" approach. Rather than instantiating multiple distinct convolutional paths, they apply learned masks to a single maximal convolutional kernel. This allows the optimization of kernel dimensions and filter counts within the memory footprint of a single convolution.


Adapting to Task Scale and Complexity

Direct search means the optimization process must natively handle the scale of real-world data and the unique geometric constraints of downstream tasks.

Data and Bottleneck Scheduling

We can also save compute by being smart about how data is fed into the search. Chen et al. coupled data scheduling with network pruning. Their method starts the search on "easy" subsets of the target dataset and progressively prunes unpromising paths from the architectural search space as harder data is introduced, preventing the prohibitive costs of full-scale training.

Alternatively, we can exploit structural properties of deep networks. Yang et al. observed that network performance is often dictated by a "vital few" bottleneck layers—the neck of an hourglass. By prioritizing the search on these critical information-flow components and efficiently filling in the remaining blocks, they achieved direct ImageNet optimization within the computational budget of traditional proxy searches.

Task-Specific Alignment

When the target task is not image classification, proxy-based NAS fails catastrophically because classification networks do not preserve the spatial resolutions required for localization or dense pixel prediction.

  • Semantic Segmentation: Li et al. and Zhang et al. demonstrated that searching directly on target segmentation datasets ensures the discovered macro-architectures are specifically tailored to the spatial and structural requirements of pixel-level prediction.
  • Visual Tracking: Marvasti-Zadeh et al. showed that direct optimization on video tracking datasets allows the search space to natively adapt to temporal object localization and scale variations.

Breaking the Cell-Stacking Paradigm

Traditional NAS forces a rigid constraint: the algorithm searches for a single, repeatable "cell" block, which is then stacked identically throughout the network. However, the optimal operations at the early layers of a network (which handle high-resolution, low-level features) are rarely the same as those at the deeper layers (which handle low-resolution, high-level semantics).

Direct large-scale search allows us to relax this constraint entirely. Zhu et al. removed the cell-stacking requirement, allowing every single layer in the target network to evolve a unique architecture with global connectivity weights. Because the search is conducted directly on the final network size, there is no mismatch between the search space and the evaluation space.

Cell-Stacking Paradigm (Proxy-dependent):
[ Search Cell A ] ---> [ Repeat Cell A ] ---> [ Repeat Cell A ]

Macro-Architecture Relaxation (Direct Search):
[ Search Layer 1 ] ---> [ Search Layer 2 ] ---> [ Search Layer 3 ]
  (Low-level ops)         (Mid-level ops)         (High-level semantic ops)

To manage the massive search space that opens up when every layer can be unique, Chen et al. modeled architecture weights as random variables using a Dirichlet distribution combined with progressive widening and pruning. This ensures that the selected features are mathematically optimized for the final model's actual depth and width. To prevent memory exhaustion during macro-search, You et al. decoupled the architecture using a hybrid structure of "Engine-cells" (which drive the search) and "Transit-cells" (which mimic the final deployment environment), reducing the number of active differentiable parameters.


Closing the Search-Evaluation Discrepancy

In classical NAS, the search phase operates on a continuous, dense super-network, but the evaluation phase operates on a pruned, sparse target network. This discrepancy means the winning architecture in the search phase might not be the winner once it is isolated and retrained from scratch.

Direct, proxyless search enables a one-stage optimization paradigm where the search and evaluation models are identical. Yang et al. utilized sparse coding to simultaneously optimize both the architecture topology and the network weights. This one-stage approach ensures that the network evaluated is exactly the network searched, eliminating the post-search pruning step.

Similarly, Gao et al. introduced minimum entropy regularization to force the architecture parameters to converge to discrete choices during the search. This allows the model to be evaluated immediately after the search phase without requiring retraining from scratch. By preserving the weights learned during the search, frameworks like the one proposed by Li et al. completely align the search phase with the final deployment environment, turning a disjointed multi-stage pipeline into a streamlined, direct optimization process.

Key Foundational Work

PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search (2020) — contributed 2 concept elements to this direction.
MTL-NAS: Task-Agnostic Neural Architecture Search Towards General-Purpose Multi-Task Learning (2020) — contributed 1 concept elements to this direction.
ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware (2019) — contributed 1 concept elements to this direction.
History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 15 papers
  2. Version 1 · 2026-10-04 · 32 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_direct_large_scale_architecture_search_da55c675,
  title = {direct large-scale architecture search},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/direct-large-scale-architecture-search/rdr_da55c675398f4f068fb2344a4ebe0596},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 2, frozen text):

@misc{lacuna_direct_large_scale_architecture_search_da55c675_v2,
  title = {direct large-scale architecture search},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/direct-large-scale-architecture-search/rdr_da55c675398f4f068fb2344a4ebe0596/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-22},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake