Progressive Expansion for Efficient Neural Architecture Search

37 concepts30 papers1 proposal

The central challenge of Neural Architecture Search (NAS) is the "space explosion" problem. The number of possible ways to connect layers and operations grows exponentially with the size of the network, making it impossible to exhaustively evaluate every candidate. Traditional methods often treat each architecture as an independent black box, training thousands of models from scratch.

Progressive expansion shifts this paradigm. Instead of searching the full space at once, these methods start with a minimal, high-performing foundation and incrementally grow the architecture. By building complexity in stages, the search process becomes a manageable curriculum, allowing models to learn fundamental features before tackling the intricate representations required for high-capacity performance.

Managing Search Complexity Through Staged Growth

The most direct way to handle a massive search space is to avoid entering it all at once. Chenxi Liu 0001 et al. demonstrated this with Progressive Neural Architecture Search (PNAS), which searches for modular cells in order of increasing complexity. By evaluating simpler architectures first, the system generates early training data to train a "surrogate" model, which then predicts the performance of more complex candidates without needing to train them.

This staged approach also solves the "depth gap"—the discrepancy between the shallow models used during search and the deep models used in final deployment. Lingxi Xie et al. showed that gradually increasing the number of layers during the search process allows the architecture to adapt to the specific gradient dynamics of deep networks. Similarly, Jin-Dong Dong et al. used progressive expansion to discover Pareto-optimal "mobile-first" architectures, ensuring that the search remains computationally tractable even when balancing multiple objectives like accuracy and latency.

Efficiency via Weight Reuse and Warm-Starting

Training a neural network is expensive; training a thousand is often prohibitive. Progressive expansion mitigates this by treating architecture design as an iterative growth problem rather than a series of independent trials.

Han Cai et al. introduced Efficient Architecture Search (EAS), which uses reinforcement learning to "grow" existing networks using function-preserving transformations. Because the new, larger network starts with the logic and weights of its predecessor, it requires significantly less training to reach peak performance. Hanzhang Hu et al. extended this concept with Petridish, which grows networks forward by adding shortcut connections. This allows for "warm-starting" the search from an existing production model, refining it as more computational resources become available rather than starting from a blank slate.

Curriculum Search and Operation Expansion

The complexity of a search space isn't just about the number of layers; it's also about the variety of operations (like convolutions, pooling, or skip connections) available at each step. Yaofo Chen et al. argued that introducing all operations simultaneously creates a "noisy" search environment. Their Curriculum Neural Architecture Search (CNAS) starts with a single candidate operation and introduces new ones in successive stages.

This staged introduction of operations ensures fair competition. For instance, Xunyu Zhu et al. found that skip connections often dominate early in differentiable searches because they are easy to optimize, leading to "collapse." By introducing operations one by one, the model stabilizes its weights for complex operations before the architectural competition becomes too intense.

Cognitive and Biological Foundations

The idea of starting simple and growing complex mirrors both human education and biological development. Petru Soviany et al. formalize this as "model-level curricula," where the gradual expansion of a network's representational power allows it to learn fundamental features before transitioning to high-capacity modeling.

This "neuroplasticity" is particularly effective in resource-constrained environments. Yebo Wu et al. proposed Developmental Federated Tuning (DEVFT), which mimics cognitive development by starting with a compact "child" submodel on edge devices and progressively growing it into an "adult" model. This strategy slashes communication costs and accelerates convergence by navigating a simpler optimization landscape in the early stages of training.

Beyond Neural Topology

The principles of incremental growth extend to other domains of automated discovery:

  • Symbolic Regression: Nan Jiang 0012 et al. build mathematical expressions by starting with a single variable and gradually incorporating others, breaking an exponential search space into polynomial-sized chunks.
  • Data Augmentation: Yu Zheng 0022 et al. construct augmentation pipelines by searching for and fixing one layer at a time, reducing search complexity from exponential to linear.
  • Optimization: Wei Ping et al. use the Frank-Wolfe algorithm to greedily add hidden units to Restricted Boltzmann Machines, allowing the data itself to determine the optimal model complexity.

By treating architecture as a living structure that grows in response to the task, researchers move away from the brute-force search of the past toward a more efficient, structured, and biologically plausible form of machine learning.

Key Foundational Work

Searching Toward Pareto-Optimal Device-Aware Neural Architectures (2018) — contributed 1 concept elements to this direction.
PPP-Net: Platform-aware Progressive Search for Pareto-optimal Neural Architectures (2018) — contributed 1 concept elements to this direction.
Efficient Architecture Search by Network Transformation (2018) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_incremental_architecture_search_d59ffb54,
  title = {incremental architecture search},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/incremental-architecture-search/rdr_d59ffb54c200489b9deac55c1fb31253},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_incremental_architecture_search_d59ffb54_v1,
  title = {incremental architecture search},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/incremental-architecture-search/rdr_d59ffb54c200489b9deac55c1fb31253/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake