Continuous Latent Spaces for Neural Architecture Optimization

This direction is split into other directions: non-differentiable hardware optimization. Its last version stays readable and citable here.

Version 1 of 161 papers64 conceptsfrozen 2026-10-04

Designing a neural network has historically been a "black art." Researchers like Barret Zoph and Quoc V. Le noted in Neural Architecture Search with Reinforcement Learning that finding the right combination of layers, kernels, and skip connections requires immense human intuition and trial-and-error.

The fundamental problem is that neural architectures are discrete directed acyclic graphs (DAGs). You cannot take the derivative of "adding a layer" or "changing a $3 \times 3$ convolution to a $5 \times 5$." This discreteness creates a "spiky," non-differentiable landscape that breaks the very tool—gradient descent—that makes deep learning work.

The Discrete Bottleneck

In standard training, we use backpropagation to wiggle continuous weights. But architectural decisions—like the number of channels or the choice of an activation function—are categorical. As Yanan Sun and colleagues explain in their survey on Evolutionary NAS, this forces researchers to rely on "black-box" optimization.

Early solutions used Genetic Algorithms, such as the NeuroGENESYS system by Steven A. Harp, or Reinforcement Learning (RL). However, these methods are computationally expensive because they treat the architecture as a fixed choice: you must sample a discrete structure, train it to completion to get a "reward," and then try again. This lack of a smooth search space makes it impossible to know which direction to move in to improve performance.

Mapping Graphs to Vectors

The core insight of recent research is to bridge this gap by mapping discrete graphs into a continuous latent space.

In Neural Architecture Optimization (NAO), Renqian Luo and Tie-Yan Liu proposed an encoder-predictor-decoder framework.

  1. Encoder: An LSTM or Graph Neural Network (GNN) takes a discrete architecture string and compresses it into a continuous vector (an embedding).
  2. Predictor: A simple function learns to predict the accuracy of an architecture based on its vector.
  3. Optimization: Because the latent space is continuous, we can perform gradient ascent on the predictor to find a "better" vector.
  4. Decoder: The optimized vector is mapped back into a new, discrete architecture.

By doing this, the search shifts from a blind walk in a discrete forest to a smooth slide down a hill.

Smoothing the Landscape with Autoencoders

The effectiveness of this approach depends entirely on how "smooth" the latent space is. If two similar architectures end up in completely different parts of the latent space, the gradients become useless.

To solve this, Shen Yan and Mi Zhang introduced arch2vec in Does Unsupervised Architecture Representation Learning Help Neural Architecture Search?. They used a Variational Graph Isomorphism Autoencoder to learn representations in an unsupervised way—simply by forcing the model to reconstruct the graph structure. This ensures that architectures with similar topologies are clustered together, creating a stable landscape for optimization.

Similarly, David Friede and Jovita Lukasik developed the Variational-Sequential Graph Autoencoder (VS-GAE). Their work, detailed in Smooth Variational Graph Embeddings, demonstrates that these embeddings allow for the prediction of performance even for architectures larger than those seen during training, effectively "extrapolating" toward better designs.

Simultaneous Weight and Structure Training

If we can represent an architecture as a continuous vector, why not train the architecture and the weights at the same time?

This is the premise of SWAT-NN, developed by Ajitesh Srivastava and Mansooreh Montazerin. In Simultaneous Weight and Architecture Optimization, they show that by embedding entire networks into a multi-scale autoencoder, one can use a single gradient descent process to find the optimal structure and parameters simultaneously. This eliminates the "train-evaluate-repeat" cycle of traditional NAS.

Beyond Gradients: Global and Mixed Optimization

While continuous embeddings solve the differentiability problem, they don't always guarantee a global optimum. Some researchers have turned to Mixed-Integer Programming (MIP) to treat architecture search as a rigorous mathematical problem.

Yilin Xie and Calvin Tsay introduced NAS-GOAT in Global optimization of graph acquisition functions, which encodes graph structures into linear constraints. This allows them to provably find the best next architecture to evaluate, rather than relying on the local movements of gradient descent.

For other non-differentiable parameters like bit-precision, researchers like Benjamin Bodner use hybrid approaches. In GradFreeBits, they alternate between gradient descent for weights and evolutionary strategies for discrete bit-widths, showing that even when a full latent space isn't available, "gradient-free" islands can still be integrated into the training loop.

Papers in this version

Showing 60 of 61 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_continuous_latent_spaces_for_neural_arch_4c8cf9a0,
  title = {Continuous Latent Spaces for Neural Architecture Optimization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/continuous-latent-spaces-for-neural-architecture-optimization/rdr_4c8cf9a0ae184a9ca8e672022919364e},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_continuous_latent_spaces_for_neural_arch_4c8cf9a0_v1,
  title = {Continuous Latent Spaces for Neural Architecture Optimization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/continuous-latent-spaces-for-neural-architecture-optimization/rdr_4c8cf9a0ae184a9ca8e672022919364e/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}