Neural Architecture Search (NAS) aims to automate the design of deep learning models, but the computational cost of training thousands of candidate architectures is often prohibitive. Differentiable Architecture Search (DARTS) attempted to solve this by using a "supernet" where all possible operations exist simultaneously, allowing the search to be performed via gradient descent.
However, DARTS and its successors frequently suffer from architectural collapse: a failure mode where the search process converges prematurely to simplistic operations, most notably skip-connections. This results in "shallow" models that perform well during the search phase but fail to generalize. Recent research has focused on replacing standard validation-based gradients with more robust performance estimators to stabilize this process.
The Skip-Connection Trap
In a differentiable search space, the algorithm chooses between operations (like convolutions, pooling, or identity mappings) by assigning them weights. The intuition is that the weights for "better" operations will increase over time.
The problem, as identified by Yaran Chen and Zixiang Ding in Stacked BNAS, is that skip-connections provide a "shortcut" for gradients. During the early stages of supernet training, skip-connections allow the loss to decrease rapidly because they don't require learning complex weights. The optimizer greedily selects these shortcuts, leading to a degenerate architecture that lacks the depth and complexity needed for high-level feature extraction.
Training Speed as a Performance Proxy
To bypass this trap, Binxin Ru, Clare Lyle, and Mark van der Wilk propose a shift in how we evaluate architectures. Instead of looking at instantaneous validation loss, they introduce Training Speed Estimation (TSE) in Speedy Performance Estimation for Neural Architecture Search.
The core idea of TSE is remarkably simple: the sum of training losses during the very first few iterations of training is a powerful predictor of an architecture's final performance.
$$TSE = \sum_{t=1}^{T} \mathcal{L}_{train}(\theta_t)$$
Why does this work?
- Learning Trajectory: By summing the early losses, TSE captures the efficiency of the learning process rather than just a snapshot of accuracy.
- Robustness to Shortcuts: Unlike validation gradients, which can be easily "fooled" by the rapid convergence of skip-connections, the training loss trajectory reveals which architectures are actually capable of absorbing information from the dataset.
- Computational Efficiency: Because TSE only requires a few iterations of training, it is significantly faster than traditional early-stopping methods.
Modeling Structural Dependencies
While TSE improves the signal used for selection, other researchers focus on the search space itself. Standard differentiable NAS treats each architectural choice as independent, but in reality, the "value" of a convolution depends on the operations surrounding it.
In Learning Latent Architectural Distribution in Differentiable Neural Architecture Search via Variational Information Maximization, Yaoming Wang and Hongkai Xiong introduce VIM-NAS. They treat the network structure as a latent representation of the dataset. By maximizing the mutual information between the architecture and the data, they incorporate structural dependencies into the search. This prevents the optimizer from making "locally" optimal choices (like a single skip-connection) that are "globally" suboptimal for the network's overall representational power.
From Days to Minutes
The shift toward these robust estimators has transformed the efficiency of NAS. Early NAS methods required thousands of GPU hours. By using training speed signals and improved search stability:
- TSE allows for accurate ranking of architectures without training them to completion, as shown by Lisa Schut and Yarin Gal in Revisiting the Train Loss.
- Stacked BNAS achieves state-of-the-art results in just 0.02 GPU days (approx. 30 minutes) by combining broad search spaces with early stopping criteria designed to catch performance collapse before it happens.
- VIM-NAS can converge in a single training epoch, reducing search time to roughly 10 minutes on a standard GPU.
By moving away from "greedy" validation gradients and toward metrics that capture the long-term learning potential of an architecture—like training speed and structural information—these methods make automated model design both stable and accessible.