Neural Architecture Search for Universal Structural Primitives

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2437 papers456 conceptsfrozen 2026-10-04

For decades, the progress of deep learning was gated by "architecture engineering"—the painstaking manual process of designing layers, skip connections, and branching patterns. Researchers like Quoc V Le and Barret Zoph fundamentally shifted this paradigm by proposing Neural Architecture Search (NAS). Instead of designing a model, we design a search space and let an optimization agent discover the best configuration within it.

The core intuition behind this field is that neural networks possess "structural primitives"—fundamental connectivity patterns—that are more efficient than those humans typically intuit. By automating the discovery of these primitives, we can find models that are not only more accurate but also orders of magnitude faster and more robust across diverse domains.

The primary challenge of NAS is the scale of the search space. A typical space can contain $10^{18}$ to $10^{390}$ possible configurations, as noted by Lichuan Xiang in their work on Hierarchical Generative Modeling.

Early methods used Reinforcement Learning (RL) where a controller (often an RNN) would sample an architecture, train it to convergence, and use the validation accuracy as a reward signal. While effective, this was computationally ruinous. To solve this, researchers developed more efficient navigation strategies:

  • Evolutionary Precision: Yukang Chen and Xinggang Wang introduced RENAS, which uses a "mutation controller" to learn which architectural changes are likely to succeed, combining the diversity of evolution with the directed learning of RL.
  • Differentiable Approximation: Methods like DATA by Xinbang Zhang and Gaofeng Meng relax the discrete search space into a continuous one. This allows the architecture itself to be optimized via backpropagation, drastically reducing search time.
  • Weight Sharing: Rather than training thousands of models from scratch, "one-shot" methods like FairNAS train a single "supernet" containing all possible paths. Xiangxiang Chu demonstrated that by enforcing "strict fairness"—ensuring every path is updated equally—we can accurately rank architectures without full training.

The Universality of Discovered Primitives

Perhaps the most surprising finding across these 348 papers is that architectures discovered for one task (like ImageNet classification) often excel at entirely different tasks. This suggests that NAS is uncovering "universal structural primitives"—connectivity motifs that provide fundamental inductive biases for processing data.

For instance, Duncan Watson-Parris showed in DENSE that the same architectural templates used for computer vision can be automatically adapted to achieve a 2-billion-fold speedup in scientific simulations for climate science and fusion energy. Similarly, Wuyang Chen found that multi-scale branching patterns discovered for human pose estimation generalize across different datasets without further tuning.

This transferability extends even to robustness. NAS-OoD demonstrates that by searching for architectures using adversarial data generators, we can discover structural motifs that are inherently more stable against distribution shifts, a property that remains consistent even when the model is moved to new, real-world environments.

While NAS aims to automate design, it has increasingly begun to incorporate human "topological elegance." Minghao Guo and Dahua Lin proposed IRLAS, which uses Inverse Reinforcement Learning to extract design principles from expert-crafted models like ResNet. This guides the search agent toward models that are not just accurate, but also "elegant"—simpler and faster to execute.

Conversely, researchers are using NAS to teach humans. By visually analyzing architecture spaces, Shixia Liu and colleagues have summarized design principles that humans can then use to build better models manually.

The LLM Era of Architecture Design

The latest frontier in this research direction involves Large Language Models (LLMs). Rather than using specialized RL controllers, researchers are now using the code-generation capabilities of LLMs to act as "neural architects."

  • EvoPrompting: David So demonstrated that LLMs can serve as evolutionary engines, performing crossover and mutation on raw architecture code to discover novel models that outperform human baselines.
  • Zero-Shot Prediction: Ganesh Jawahar showed that LLMs possess an inherent "understanding" of neural network performance. By using LLM-based predictors to initialize a search, we can cut the computational cost of NAS by 50%.
  • Autonomous Pipelines: Systems like NN-Caption by Radu Timofte allow LLMs to autonomously assemble components like attention blocks and squeeze-and-excitation layers into functional, high-performing architectures for complex tasks like image captioning.

Ultimately, this research direction suggests that the "best" neural network is rarely a simple stack of layers. Instead, it is a complex, often counter-intuitive arrangement of connections that can be systematically discovered through automated search, meta-learning, and increasingly, the distilled architectural knowledge within large-scale generative models.

Papers in this version

Showing 60 of 437 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 422 papers
  2. Version 1 · 2026-10-04 · 437 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_architectural_transferability_0dcf17fb,
  title = {architectural transferability},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/architectural-transferability/rdr_0dcf17fb39794f4aa4905d2e6ad136ad},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_neural_architecture_search_for_universal_0dcf17fb_v1,
  title = {Neural Architecture Search for Universal Structural Primitives},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/neural-architecture-search-for-universal-structural-primitives/rdr_0dcf17fb39794f4aa4905d2e6ad136ad/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}