Differentiable Architecture Search (DARTS) revolutionized the field of Neural Architecture Search (NAS) by turning a discrete, combinatorial search problem into a continuous optimization task. Instead of training thousands of candidate models from scratch, DARTS uses a "supernet" where every possible operation (convolutions, pooling, skip-connections) exists in parallel. By learning a set of mixing weights—the architecture parameters—the algorithm can theoretically "relax" the search space and find the best model using simple gradient descent.
However, DARTS has a notorious "dark side." As the search progresses, it often suffers from performance collapse: the algorithm converges to degenerate solutions dominated by parameter-free operations, particularly skip-connections (identity mappings). These models are fast to train during the search phase but lack the representational depth to perform well on real-world tasks.
The Unfair Advantage of Skip Connections
The root of performance collapse lies in an optimization bias. Skip connections allow gradients to flow through the supernet with minimal resistance. During the early stages of search, an identity mapping provides a more immediate reduction in loss than a complex, uninitialized convolution.
Xiangxiang Chu and colleagues in Fair DARTS identify this as an "unfair advantage." Because the search process is greedy, it over-favors these "easy" operations, leading to a feedback loop where skip connections are strengthened, further starving the convolutional layers of meaningful gradients. This results in a shallow, non-functional architecture once the search is discretized.
Balancing the Search Space
To counter this bias, researchers have introduced mechanisms to normalize the competition between operations. Huiqun Wang et al. proposed iDARTS, which uses Node Normalization to balance signal strength across different operations. By ensuring that no single operation can dominate the gradient flow simply due to its mathematical structure, the search remains competitive for longer.
Another approach involves structural grouping. Guilin Li and the authors of StacNAS suggest that collapse happens when similar operators compete against each other. By using hierarchical operator clustering, they prevent redundant operations from cannibalizing the search, ensuring a more stable and consistent convergence.
Decoupling Weights and Architecture
A significant source of instability in DARTS is the bi-level optimization problem: the algorithm tries to update network weights and architecture parameters simultaneously. If the weights haven't converged, the architecture gradients are noisy and unreliable.
Miao Zhang et al. addressed this in their version of iDARTS by using the Implicit Function Theorem (IFT). By calculating stochastic implicit gradients, they decouple the architecture search from the specific optimization path of the weights. This allows the search to "see through" the temporary training benefits of skip connections and focus on the long-term representational power of the operations.
Similarly, Erik Bodin et al. argue in Making Differentiable Architecture Search less local that the optimizer often gets trapped in poor local minima. They propose a hybrid local-global optimization scheme that allows the search to escape these traps, moving beyond the simplistic structures that characterize performance collapse.
From Point Optimization to Distribution Learning
Perhaps the most robust shift in this research direction is moving away from finding a single "perfect" wiring to learning a distribution over architectures.
Jilin Hu and Miao Zhang introduced BaLeNAS, which reformulates NAS as a Bayesian distribution learning problem. By treating the architecture as a random variable, the method naturally encourages exploration and provides implicit regularization. This prevents the search from prematurely collapsing into a single suboptimal point.
This probabilistic perspective is shared by Chao Xue et al. in GibbsNAS, which uses Gibbs sampling to navigate the search space. By accounting for uncertainty, these methods avoid the rigid, deterministic traps that lead to skip-connection dominance. Xingchen Wan et al. take this further with ANASOD, optimizing the ratio of operations (e.g., the percentage of 3x3 convolutions) rather than the specific connections, which significantly stabilizes the search.
Harmonizing the Network
Finally, some researchers look at the global consistency of the discovered network. Sajad Movahedi and the authors of $\Lambda$-DARTS introduced layer alignment. They observed that in collapsed models, different layers often disagree wildly on which operations are useful. By forcing different cells to harmonize their operation selections, $\Lambda$-DARTS prevents the search from devolving into a collection of disconnected, shallow identity mappings, ensuring the final model retains functional depth.