Differentiable Architecture Search (DARTS) transformed neural architecture search (NAS) from a month-long reinforcement learning task into a few-day gradient descent problem. By relaxing the discrete choice of operations (like $3\times3$ convolutions or max pooling) into a continuous "supernet," researchers could use standard backpropagation to learn the architecture parameters ($\alpha$) alongside model weights ($w$).
However, this efficiency came with a fatal flaw: performance collapse. Left to its own devices, the search process often converges to "degenerate" architectures composed almost entirely of skip connections (identity mappings). These models are easy for the optimizer to train but lack the representational capacity to perform well on real-world tasks.
The Unfair Advantage of Skip Connections
To understand why DARTS collapses, we must look at the optimization landscape of the supernet. In a typical DARTS setup, the output of an edge between two nodes is a weighted sum of candidate operations, normalized by a softmax function.
As Xiangxiang Chu, Bo Zhang, and their colleagues argue in Fair DARTS, the softmax creates a zero-sum competition. Operations that facilitate rapid gradient flow—specifically skip connections—gain an early lead. Because they don't have weights to train, skip connections provide a "clean" path for gradients to reach earlier layers. In contrast, a convolution must learn its weights before it can contribute meaningfully to reducing the loss. By the time the convolution has matured, the architecture parameters have already locked onto the skip connection, suppressing the more complex operation.
This is not a sign of architectural superiority, but of optimization speed. Pan Zhou and Steven Chu-Hong Hoi demonstrate in PR-DARTS that skip connections exhibit faster initial convergence rates, creating a bias that favors short-term stability over long-term performance.
Overfitting the Supernet
Another perspective on collapse involves the bi-level optimization process itself. Hanwen Liang and Shifeng Zhang identify overfitting as a primary culprit in DARTS+.
During the search, the model weights $w$ are updated on the training set, while architecture parameters $\alpha$ are updated on the validation set. As the weights $w$ begin to overfit the training data, the validation signal becomes noisy. The architecture optimizer reacts by selecting skip connections to "bypass" these overfitted layers, seeking cleaner feature representations from earlier in the network. This results in shallow, ineffective final models.
Leveling the Playing Field
To fix the collapse, researchers have moved toward "robustifying" the search process by removing the unfair advantages of simple operations.
- Decoupling Competition: Fair DARTS replaces the competitive softmax with independent sigmoid activations. This allows multiple operations to "cooperate" during training, preventing skip connections from prematurely suppressing convolutions.
- Factoring Out Gradients: In DARTS-, Xiangxiang Chu introduces an auxiliary skip connection that exists outside the architecture selection. This "factors out" the gradient flow benefit, forcing the candidate skip connections to compete based on their actual representational value rather than their ability to smooth the loss landscape.
- Early Stopping: DARTS+ introduces simple criteria to stop the search before the supernet weights overfit, effectively catching the architecture at its peak diversity before it collapses into identity mappings.
Rethinking Selection Rules
Perhaps the most surprising insight in this field is that the architecture parameters ($\alpha$) themselves might be the wrong thing to look at.
In Rethinking Architecture Selection in Differentiable NAS, researchers found that the magnitude of $\alpha$ does not necessarily correlate with an operation's contribution to final accuracy. They proposed a perturbation-based selection rule: instead of picking the operation with the largest weight, they measure how much the validation performance drops when an operation is removed. This approach reveals that many skip connections with high weights are actually redundant, while "weaker" convolutions are doing the heavy lifting.
Similarly, Lichuan Xiang and Łukasz Dudziak show in Zero-Cost-PT that "zero-cost proxies"—metrics that evaluate a network's potential without any training—can be used to prune the search space and avoid skip-connection dominance entirely.
Beyond Image Classification
While much of this work focused on CIFAR-10, the "skip-connection bias" is a broader phenomenon. Mikhail Khodak and Ameet Talwalkar demonstrate in NAS-Bench-360 that this inductive bias, while sometimes helpful in computer vision, can lead to catastrophic failures in scientific domains like fluid dynamics or genomics, where identity mappings are less useful.
The evolution of this research direction shows a shift from treating NAS as a pure optimization problem to treating it as a delicate balancing act between optimization speed and representational capacity. By understanding the "why" behind performance collapse, researchers have moved from fragile, heuristic-heavy searches to robust methods that can reliably discover complex architectures in minutes rather than days.