Bridging the Depth Gap in Differentiable Architecture Search

This direction is merged into another direction: operation selection bias, search-evaluation depth gap. Its last version stays readable and citable here.

Version 1 of 117 papers20 conceptsfrozen 2026-10-04

In Neural Architecture Search (NAS), we want to find the best "cell"—a small motif of operations—to build a deep neural network. However, searching for this cell directly on a deep network is computationally expensive. To save time, researchers typically use a proxy task: they search for the best cell using a shallow network (e.g., 8 layers) and then stack that same cell many more times (e.g., 20 layers) for the final evaluation.

This creates the depth gap. As Lingxi Xie, Xin Chen, and Qi Tian argue in Progressive Differentiable Architecture Search: Bridging the Depth Gap Between Search and Evaluation, an architecture that excels at a depth of 8 may be suboptimal at a depth of 20. This discrepancy is the primary reason why many NAS methods fail to generalize from the search phase to real-world deployment.

The Intuition of Topology Collapse

Why does depth change which architecture is "best"? In a shallow network, the primary challenge is often just getting the signal to flow. Differentiable search methods like DARTS tend to favor operations that facilitate gradient flow, such as skip connections (identity mappings).

In a shallow proxy, these skip connections look highly effective because they make optimization easy. However, as the search progresses, the network often suffers from topology collapse: the search algorithm fills the cells with skip connections, leaving no room for the complex convolutions needed for high representational capacity. When these "hollow" cells are stacked into a deep network, the final model lacks the depth of features required for complex tasks like ImageNet classification.

Progressive Growth as a Bridge

The most direct way to bridge this gap is to make the search environment look more like the evaluation environment. Lingxi Xie and colleagues introduced P-DARTS, which gradually increases the depth of the search network during the optimization process.

By starting shallow and ending deep, the algorithm "learns" to handle the gradient challenges of deeper structures. To keep the computational cost manageable, P-DARTS pairs this growth with search space pruning—dropping the least promising operations as the network gets deeper. This ensures that by the end of the search, the cell has been optimized for the actual conditions it will face at deployment.

Structural and Probabilistic Consistency

Other researchers have focused on making the search itself more robust to the depth gap without necessarily training massive networks.

Why It Works

The common thread across these papers—from Bo Lyu's TND-NAS to Huafeng Qin's EM-DARTS—is the recognition that ranking consistency is more important than absolute accuracy during search.

If Architecture A is better than Architecture B at depth 8, but worse at depth 20, a shallow search is fundamentally deceptive. By bridging the depth gap—whether through progressive growth, structural mimicking, or Bayesian regularization—these methods ensure that the "winner" of the search phase remains the winner at deployment. This alignment transforms NAS from a fragile optimization trick into a reliable tool for discovering high-performance deep learning models.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_bridging_the_depth_gap_in_differentiable_8742dbd7,
  title = {Bridging the Depth Gap in Differentiable Architecture Search},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/bridging-the-depth-gap-in-differentiable-architecture-search/rdr_8742dbd7c89343e5b9872dec98da2260},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_bridging_the_depth_gap_in_differentiable_8742dbd7_v1,
  title = {Bridging the Depth Gap in Differentiable Architecture Search},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/bridging-the-depth-gap-in-differentiable-architecture-search/rdr_8742dbd7c89343e5b9872dec98da2260/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}