Neural Architecture Search with Reinforcement Learning

38 concepts34 papers1 proposal

Designing a neural network architecture is fundamentally a discrete combinatorial optimization problem. When choosing the number of layers, kernel sizes, or skip connections, there is no continuous surface to compute gradients against; you cannot take the derivative of "adding a residual connection." Historically, this forced researchers to rely on human intuition and exhaustive trial-and-error.

Reinforcement Learning (RL) changes this paradigm by framing architecture design as a sequential decision-making process. By treating the search space as an environment and the architectural choices as actions, we can train an agent—a controller—to discover high-performing topologies. The controller learns a probability distribution over the structural possibilities, shifting its choice probabilities toward configurations that maximize the performance of the generated models.

Framing Architecture Generation as an MDP

To understand why this works, imagine the construction of a neural network as a game played by an RL agent. The agent makes a sequence of moves, where each move corresponds to selecting a specific structural component (e.g., a convolution operation, a pooling layer, or a pointer to a previous layer for a skip connection).

In the foundational work by Barret Zoph et al. in Learning Transferable Architectures for Scalable Image Recognition, this sequence of decisions is generated by a recurrent neural network (RNN) acting as the controller.

[ Controller (RNN) ] ───Samples Tokens───> [ Architecture Specification ]
        ▲                                                │
        │                                             Builds & Trains
  Policy Gradient Update                                 │
        │                                                ▼
  [ Reward (Validation Acc) ] <────Evaluates──── [ Child Network ]

The process operates in a loop:

  1. Generation: The controller samples a sequence of tokens that specifies the architecture of a "child network."
  2. Evaluation: The child network is built and trained on a target dataset to obtain its validation accuracy.
  3. Optimization: The validation accuracy serves as a scalar reward signal. The controller is updated using policy gradient methods (like REINFORCE) to increase the log probability of the sampled actions if the reward is high, and decrease it if the reward is low.

This sequential framing allows the agent to capture complex structural dependencies. For instance, the choice of operation at layer $L$ can be conditioned on the choices made at layer $L-1$. Rather than optimizing each layer in isolation, the controller learns how combinations of layers interact.

While policy gradient methods dominate, value-based reinforcement learning offers an alternative perspective. Bowen Baker et al. demonstrated in Designing Neural Network Architectures using Reinforcement Learning that Q-learning can navigate this space by treating the selection of a layer configuration as a transition between states in a Markov Decision Process (MDP), using an $\epsilon$-greedy strategy to balance exploration and exploitation.

Bypassing the Computational Bottleneck

The primary drawback of the standard RL approach is its extreme computational cost. Training thousands of child networks from scratch to obtain a single reward signal can require thousands of GPU hours. To make RL-driven architecture search practical, researchers developed mechanisms to reuse weights and bypass full training cycles.

Weight Sharing via Supernets

Instead of treating every sampled architecture as an isolated model, we can view the entire search space as a single, massive Directed Acyclic Graph (DAG)—a supernet. In this formulation, every possible architecture is simply a sub-graph or a specific path through this master graph.

Hieu Pham 0001 et al. introduced this concept in Faster Discovery of Neural Architectures by Searching for Paths in a Large Model through Efficient Neural Architecture Search (ENAS). The child models share a common set of parameters for the operations residing on the nodes of the DAG.

During search, the training alternates between two phases:

  • Child Weight Update: A path is sampled by the controller, and the weights of that specific sub-graph are updated via standard gradient descent on a mini-batch of training data.
  • Controller Policy Update: The controller samples a path, the sub-graph is evaluated on a validation batch using the current shared weights, and the resulting accuracy updates the controller's policy.

Because the sub-graphs do not need to be trained from scratch, the evaluation step becomes a cheap forward pass, cutting the search time by up to three orders of magnitude. This parameter-sharing paradigm has been successfully extended to specialized domains, such as biometric identification tasks by Shaojiang Deng et al. in GRU-based Neural architecture search for finger-Vein identification.

Network Morphism

Another way to avoid training from scratch is to start with an already optimized, high-performing network and incrementally "grow" it. Han Cai et al. proposed this in Efficient Architecture Search by Network Transformation.

Instead of generating a full sequence of tokens from scratch, the RL agent learns to select function-preserving transformations (such as inserting a layer initialized as an identity mapping, or widening a channel via weight replication). Because the transformation preserves the network's exact mathematical function, the child network inherits the baseline performance immediately, allowing the RL agent to explore structural expansions with minimal training overhead.

Tailoring Search Spaces for Diverse Domains

The intuition behind RL-based architecture search extends far beyond standard convolutional or recurrent blocks. By modifying the action space and the reward function, the same underlying mechanism can optimize highly specialized structures.

Graph Neural Networks (GNNs)

GNNs are notoriously sensitive to structural choices like aggregation functions (mean, max, sum) and attention mechanisms. Furthermore, deep GNNs suffer from "over-smoothing," where node representations become indistinguishable.

To automate this, Yang Gao 0024 et al. developed Graph Neural Architecture Search (GraphNAS), which uses an RNN controller to sequentially generate GNN design choices. To handle the complex performance landscapes of graphs, Kaixiong Zhou et al. proposed in Auto-GNN: Neural architecture search of graph neural networks a conservative controller that restricts exploration to single-dimension modifications of a high-performing parent model, isolating the impact of specific structural changes.

To combat over-smoothing explicitly, Guosheng Feng et al. utilized Deep Q-Learning in Search for deep graph neural networks to navigate a search space containing flexible residual connections and identity mappings.

Input Embeddings and Memory Constraints

In large-scale recommendation systems, the primary architectural challenge is not the depth of the network, but the size of the sparse embedding tables, which consume massive amounts of memory.

Manas Joglekar et al. addressed this in Neural Input Search for Large Scale Recommendation Models by using an RL controller to sample mixed-size embedding dimensions for different features. Crucially, the reward function is reshaped to balance predictive accuracy against a strict memory budget:

$$\text{Reward} = \text{Accuracy} - \lambda \cdot \max(0, \text{Memory} - \text{Budget})$$

This penalty forces the policy gradient to favor configurations that compress low-frequency features while allocating larger embedding dimensions only to high-impact features. A similar off-policy RL framework was designed by Yunke Qu et al. in Scalable Dynamic Embedding Size Search for Streaming Recommendation to dynamically adjust embedding allocations in streaming environments.

Advanced Search Topologies

As the field matured, researchers realized that simple sequential token generation could be improved by structuring the search policy itself more intelligently.

Representing an architecture as a flat sequence of tokens can obscure long-range structural dependencies, such as the interaction between early downsampling layers and later upsampling cells. Xiu Su et al. addressed this in Prioritized Architecture Sampling With Monto-Carlo Tree Search by modeling the search space as a tree.

By using Monte-Carlo Tree Search (MCTS), the algorithm can backpropagate the validation loss of a sampled sub-network up through the specific branches that generated it. This allows the tree structure to explicitly memorize and prioritize high-performing topological patterns across non-adjacent layers.

                  [ Root Search State ]
                       /        \
                      /          \
         [ Choice: Conv 3x3 ]   [ Choice: Conv 5x5 ]
                /                      \
      [ Choice: Skip ]           [ Choice: No Skip ]
             │                          │
      (Evaluate & Backpropagate Validation Loss Up the Tree)

Hybridizing Evolution and RL

Evolutionary algorithms are excellent at maintaining a diverse population of candidate architectures, preventing the search from getting stuck in local minima, but their mutation steps are purely stochastic.

Gaofeng Meng et al. merged these two paradigms in RENAS: Reinforced Evolutionary Neural Architecture Search. Instead of using random mutations to alter a network, an RL controller is trained to act as the mutation operator. The controller analyzes a parent network and predicts which specific structural change (e.g., swapping an operation or adding a connection) is most likely to yield a performance gain, transforming a random walk into a directed optimization process.

Context-Aware and Multi-Objective Policies

Traditional NAS algorithms suffer from a lack of transferability; a policy trained to find models for one dataset must start from scratch when given a new task. To solve this "goldfish memory" problem, Xin Chen et al. developed CATCH: Context-Based Meta Reinforcement Learning for Transferrable Architecture Search, which conditions the RL controller on a latent task vector describing the dataset's characteristics. This allows the policy to immediately generalize its structural insights to unseen domains.

Furthermore, modern production environments require balancing multiple competing objectives simultaneously. In AutoML for Large Capacity Modeling of Meta's Ranking Systems, Hang Yin et al. utilized an RL agent guided by a lightweight performance surrogate to navigate complex multi-objective spaces under tight production timelines.

Similarly, Yu Zhou et al. demonstrated in HM3: Hierarchical Multi-Objective Model Merging for Pretrained Models that an actor-critic RL strategy can navigate the exponential space of layer combinations to merge pretrained models, mapping user task preferences directly to optimal architectural paths without requiring any original data or retraining.

Key Foundational Work

Evolving Learning for Analysing Mood-Related Infant Vocalisation (2018) — contributed 2 concept elements to this direction.
Learning Transferable Architectures for Scalable Image Recognition (2018) — contributed 1 concept elements to this direction.
A Flexible Approach to Automated RNN Architecture Generation (2018) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_reinforcement_learning_for_architecture__9712977f,
  title = {reinforcement learning for architecture search},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/reinforcement-learning-for-architecture-search/rdr_9712977fdd894481b4419c2074009999},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_reinforcement_learning_for_architecture__9712977f_v1,
  title = {reinforcement learning for architecture search},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/reinforcement-learning-for-architecture-search/rdr_9712977fdd894481b4419c2074009999/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake