Automated Discovery of Optimal Neural Network Architectures

25 concepts24 papers1 proposal

For years, breakthroughs in deep learning were largely driven by manual architectural engineering. Human experts spent countless hours experimenting with different combinations of layer types, residual connections, and hyperparameter configurations to find topologies that maximized predictive performance. However, as noted by Bei Liu et al. in DF-ResNet: Boosting Speaker Verification Performance with Depth-First Design, determining the optimal balance of structural dimensions like network depth and width remains an open, highly unprincipled challenge when relying solely on human intuition.

To overcome these limitations, the field is undergoing a fundamental paradigm shift. As synthesized by Ke Zhang et al. in 图像分类的深度卷积神经网络模型综述 and echoed by Hirokatsu Kataoka in The Present and Future AI Technologies : Development and Applications in Image/Video Recognition and Natural Language Processing, research focus is transitioning away from manual layer configuration toward Neural Architecture Search (NAS) and Automated Machine Learning (AutoML). Instead of designing a single static network, researchers now design algorithmic frameworks that automatically discover optimal, task-specific network configurations within a predefined search space.

From Manual Engineering to Search Spaces

The core intuition behind why NAS works lies in framing network design as a structured optimization problem. Rather than guessing which layer should follow another, the human designer defines a bounded search space. This space acts as a blueprint containing all valid structural ingredients.

According to Jian Cheng 0004 et al. in A Hybrid Neural Architecture Search Algorithm Optimized via Lifespan Particle Swarm Optimization for Coal Mine Image Recognition, a typical search space encompasses variations in:

  • Layer types (e.g., standard convolutions, dilated convolutions, self-attention blocks)
  • Activation functions and pooling operations
  • Macro-architectural parameters such as total network depth, width, and routing topology

By defining these boundaries, the problem shifts from an infinite creative exercise to a discrete or continuous mathematical search over a combinatorial space of operations, as detailed by Kevin Duh and Xuan Zhang in AutoML for NLP.

Once the search space is established, an optimization algorithm navigates it to identify high-performing candidates. Different search strategies offer distinct trade-offs in efficiency and stability:

  1. Reinforcement Learning and Evolutionary Strategies: Early automated synthesis methods utilized reinforcement learning controllers or evolutionary algorithms to iteratively sample architectures, evaluate their performance, and update the selection policy, as reviewed by Wenhan Xia et al. in INVITED: Efficient Synthesis of Compact Deep Neural Networks. Similarly, population-based heuristics like Particle Swarm Optimization can effectively explore complex structural spaces under noisy environmental conditions (Jian Cheng 0004 et al.).
  2. Differentiable Search: To avoid the massive computational cost of training thousands of candidate networks from scratch, methods like Differentiable Architecture Search (DARTS) relax the discrete search space into a continuous one. This allows the architecture itself to be optimized via gradient descent alongside the network weights. However, in DARTS for Inverse Problems: a Study on Stability, Jonas Geiping et al. caution that one-shot search metrics can suffer from significant performance instability, meaning that careful validation and multiple search runs remain critical for specialized domains.

Multi-Objective Optimization for Edge Deployment

A primary driver behind the adoption of NAS is the need to deploy deep learning models on resource-constrained edge devices, such as mobile phones, drones, or embedded sensors. Human engineers struggle to manually balance the trade-off between a model's accuracy and its physical execution costs.

NAS solves this by treating network design as a multi-objective optimization problem. As explained by Abhinav Goel and Yung-Hsiang Lu in A Survey of Methods for Low-Power Deep Learning and Computer Vision, automated search algorithms can integrate hardware-specific constraints directly into the fitness function.

┌────────────────────────────────────────────────────────┐
│                  Automated Search Loop                 │
│                                                        │
│   ┌─────────────────┐         ┌────────────────────┐   │
│   │ Candidate       │         │ Multi-Objective    │   │
│   │ Architecture    │────────>│ Evaluation         │   │
│   │ Selection       │         │                    │   │
│   └─────────────────┘         └────────────────────┘   │
│            ▲                             │             │
│            │                             ▼             │
│   ┌─────────────────┐         ┌────────────────────┐   │
│   │ Optimization    │         │  Task Accuracy     │   │
│   │ Strategy Update │<────────│  + Hardware Cost   │   │
│   │ (RL, GA, PSO)   │         │  (Latency, Power)  │   │
│   └─────────────────┘         └────────────────────┘   │
└────────────────────────────────────────────────────────┘

By evaluating candidates based on both task performance and hardware metrics (such as latency, memory footprint, and power consumption), NAS discovers Pareto-optimal architectures tailored to specific deployment environments (Tianyu Wang et al. in Deploying AI on Edge: Advancement and Challenges in Edge Intelligence). Gaurav Menghani notes in Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Better that combining NAS with Hyper-Parameter Optimization (HPO) allows frameworks to simultaneously tune the macro-structure and the training parameters, maximizing efficiency within a unified ecosystem (Marius Lindauer in Hands-On Automated Machine Learning Tools: Auto-Sklearn and Auto-PyTorch).

Domain Adaptation and Open Challenges

The utility of automated architecture discovery spans diverse data modalities and complex tasks:

Despite these clear advantages, automated architecture discovery is not a silver bullet. In Xiaomi's Submissions for IWSLT 2020 Open Domain Translation Task, Yuhui Sun et al. point out that NAS-discovered architectures do not yet consistently or definitively outperform highly optimized, human-designed baselines across every domain. Systematic comparisons and better integration methodologies remain necessary to seamlessly incorporate automated structural discovery into standard deep learning workflows.

Key Foundational Work

INVITED: Efficient Synthesis of Compact Deep Neural Networks (2020) — contributed 2 concept elements to this direction.
Evolved Speech-Transformer: Applying Neural Architecture Search to End-to-End Automatic Speech Recognition (2020) — contributed 1 concept elements to this direction.
Hands-On Automated Machine Learning Tools: Auto-Sklearn and Auto-PyTorch (2019) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_neural_architecture_search_ce5fb1ac,
  title = {neural architecture search},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/neural-architecture-search/rdr_ce5fb1ac452c4a47aa57f9d0d11137bc},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_neural_architecture_search_ce5fb1ac_v1,
  title = {neural architecture search},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/neural-architecture-search/rdr_ce5fb1ac452c4a47aa57f9d0d11137bc/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake