Neural Architecture Search with Reinforcement Learning

This direction is merged into another direction: computational cost of architecture search, reinforcement learning for architecture search. Its last version stays readable and citable here.

Version 1 of 157 papers63 conceptsfrozen 2026-10-04

Designing a high-performing neural network has traditionally been a "black art." It requires years of expertise and a massive amount of trial and error to decide how many layers to use, which connections to make, and which activation functions to apply. Researchers like Barret Zoph and Quoc V Le proposed a shift in this paradigm: instead of designing the network ourselves, we can train a "meta-model" to design it for us.

This approach, known as Neural Architecture Search (NAS), treats the design of a neural network as a sequential decision-making problem. By using reinforcement learning (RL), an automated agent can navigate the vast space of possible architectures to discover structures that often outperform those designed by humans.

The Controller and Child Feedback Loop

The core of this idea is a feedback loop between two networks: a controller and a child network.

In the foundational work Neural Architecture Search with Reinforcement Learning, the controller is typically a Recurrent Neural Network (RNN). The controller acts as an agent that "samples" an architecture by generating a sequence of tokens. These tokens represent hyperparameters like filter sizes, strides, and the number of layers.

Once the controller proposes an architecture, a "child network" is built and trained on a real-world task. The performance of this child network—usually its validation accuracy—is then sent back to the controller as a reward signal. Because the process of training a network is non-differentiable, the controller uses policy gradient methods (like REINFORCE) to update its parameters. Over time, the controller learns to assign higher probabilities to architectural configurations that yield superior performance.

From Global Search to Modular Cells

Searching for an entire network from scratch is computationally expensive and difficult to scale to large datasets like ImageNet. To solve this, Jonathon Shlens, Vijay Vasudevan, and their colleagues developed NASNet.

Instead of searching for the whole structure, the controller searches for a small "cell" or building block on a proxy dataset (like CIFAR-10). Once an optimal cell is found, it is stacked multiple times to create a larger, deeper model. This "transferable" approach allowed automated designs to achieve state-of-the-art accuracy on ImageNet while being significantly more efficient than human-engineered models.

Efficiency through Weight Sharing

The primary bottleneck of early NAS was the need to train thousands of child networks from scratch. Hieu Pham and Melody Y. Guan introduced Efficient Neural Architecture Search (ENAS) to address this in 2018.

ENAS forces all candidate child models to share weights within a single large Directed Acyclic Graph (DAG). Instead of training every new architecture from zero, the controller simply selects different paths through this "supernet." This innovation slashed the computational cost of NAS by up to 1,000x, allowing researchers to discover high-performing architectures in hours rather than days.

Beyond Accuracy: Hardware and Domain Constraints

As the field matured, the reward signal evolved from simple accuracy to multi-objective optimization. Mingxing Tan and Bo Chen introduced MnasNet, which incorporated real-world latency measurements from mobile devices into the RL reward function. This ensured that the discovered models were not just accurate, but also fast enough to run on edge hardware.

The flexibility of the RL-based search has allowed NAS to be applied to highly specialized domains where human intuition is limited:

  • Activation Functions: Prajit Ramachandran used NAS to discover Swish, a smooth non-monotonic activation function that often outperforms ReLU.
  • Privacy: Anda Cheng developed DPNAS to find "privacy-friendly" architectures that maintain high utility under Differential Privacy constraints.
  • Graph Networks: Yang Gao and Peng Zhang extended the search to Graph Neural Networks, automating the selection of message-passing and aggregation functions.

Why Reinforcement Learning Works for Design

The success of RL in this space stems from its ability to navigate discrete, non-differentiable search spaces. Unlike gradient-based optimization, which requires a smooth landscape, RL agents can learn the "logic" of a good architecture. By treating model design as a sequence of actions, the controller learns complex dependencies—such as which activation functions pair best with specific layer depths—that are often counter-intuitive to human designers.

As demonstrated by frameworks like AutoGAN and AutoCaption, this automated search process consistently finds "pearls" in the design space that human experts might never have considered.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_neural_architecture_search_with_reinforc_a98dc38f,
  title = {Neural Architecture Search with Reinforcement Learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/neural-architecture-search-with-reinforcement-learning/rdr_a98dc38f049041518c0d02d84636f147},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_neural_architecture_search_with_reinforc_a98dc38f_v1,
  title = {Neural Architecture Search with Reinforcement Learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/neural-architecture-search-with-reinforcement-learning/rdr_a98dc38f049041518c0d02d84636f147/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}