Automated Discovery of Neural Architectures

This direction is split into other directions: topology-evolving neuroevolution, evolutionary architecture search, evolutionary neural architecture search. Its last version stays readable and citable here.

Version 1 of 1334 papers403 conceptsfrozen 2026-10-04

For years, designing a high-performing neural network was considered a "black art." Researchers like Kevin Duh and Hao Qin noted that manual tuning of hyperparameters—from layer depth to vocabulary preprocessing—was a tedious, intervention-heavy process. The core idea of Neural Architecture Search (NAS) and Neuroevolution is to treat the design of the network itself as an optimization problem, allowing algorithms to discover structures that human intuition might never consider.

From Manual Design to Search Spaces

The fundamental shift in this research direction is moving away from engineering specific layers and toward engineering search spaces. As Ke Zhang and colleagues explain in their survey of image classification, the focus has transitioned from manual structural engineering to defining the boundaries within which an automated agent can explore.

In this framework, an architecture is represented in a machine-readable format—often a string or a graph. Ming Gong and Linjun Shou demonstrated this with NeuronBlocks, which treats neural components like Lego bricks. By standardizing these building blocks, researchers can use optimization algorithms to plug them together in novel configurations.

The Evolutionary Mechanism

One of the most prominent methods for navigating these search spaces is the use of Evolutionary Algorithms. In this approach, a population of candidate architectures (the "genotypes") is evaluated based on their performance on a task (the "fitness").

  1. Encoding: Architectures are encoded into binary strings or chromosomes. Lingxi Xie and Alan Yuille in Genetic CNN showed how bit-wise mutations and crossovers could represent complex connectivity patterns.
  2. Selection and Mutation: The best-performing models are "bred" to create a new generation. Yanan Sun developed CNN-GA, which uses variable-length encoding to allow networks to grow or shrink adaptively, discovering the optimal depth and width without human intervention.
  3. Refinement: Over generations, the population converges on highly efficient structures. This has been applied to everything from camera calibration to heart disease prediction.

Reinforcement Learning and Controllers

An alternative to evolution is using a "controller" model—often a Recurrent Neural Network (RNN)—to propose new architectures. Quoc V Le and Barret Zoph pioneered Neural Architecture Search with Reinforcement Learning, where the controller is rewarded when its proposed "child" models achieve high validation accuracy. This creates a feedback loop where the AI learns the "grammar" of effective neural design.

Why It Works: Breaking Human Bias

The primary reason automated search succeeds is its ability to discover non-intuitive topologies. Human designers tend to favor symmetry, linear stacks, and standard skip-connections. However, automated methods often find "messy" but highly effective structures.

  • Unconventional Gates: Stephen Merity and Richard Socher found that high-performing RNNs often omit traditional output gates, a design choice that contradicts standard LSTM principles.
  • Complex Connectivity: Genetic CNN discovered multi-path connectivity patterns that outperformed the linear structures of VGGNet.
  • Task-Specific Efficiency: In real-world challenges like image denoising or super-resolution, NAS identifies specialized blocks that maximize the receptive field while minimizing computational footprint—balancing accuracy with inference speed in ways manual tuning rarely achieves.

Multi-Objective Optimization

Modern automated search doesn't just optimize for accuracy. As Kevin Duh demonstrated using Covariance Matrix Adaptation-Evolution Strategy (CMA-ES), these systems can perform multi-objective optimization. By including decoding speed or model size in the fitness function, the search process can discover the "sweet spot" for specific hardware constraints, a theme echoed in the AIM 2019 Challenge on Constrained Super-Resolution.

Ultimately, as Hirokatsu Kataoka suggests, the field is moving toward a future where the researcher's role is to define the ethics and constraints of the system, while the machine discovers the optimal mathematical path to the solution.

Papers in this version

Showing 60 of 334 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_automated_discovery_of_neural_architectu_3ab83a6c,
  title = {Automated Discovery of Neural Architectures},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/automated-discovery-of-neural-architectures/rdr_3ab83a6ce0b543928f5d0bebc46137fc},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_automated_discovery_of_neural_architectu_3ab83a6c_v1,
  title = {Automated Discovery of Neural Architectures},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/automated-discovery-of-neural-architectures/rdr_3ab83a6ce0b543928f5d0bebc46137fc/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}