In Neural Architecture Search (NAS), we want to find the best "cell"—a small motif of operations—to build a deep neural network. However, searching for this cell directly on a deep network is computationally expensive. To save time, researchers typically use a proxy task: they search for the best cell using a shallow network (e.g., 8 layers) and then stack that same cell many more times (e.g., 20 layers) for the final evaluation.
This creates the depth gap. As Lingxi Xie, Xin Chen, and Qi Tian argue in Progressive Differentiable Architecture Search: Bridging the Depth Gap Between Search and Evaluation, an architecture that excels at a depth of 8 may be suboptimal at a depth of 20. This discrepancy is the primary reason why many NAS methods fail to generalize from the search phase to real-world deployment.
The Intuition of Topology Collapse
Why does depth change which architecture is "best"? In a shallow network, the primary challenge is often just getting the signal to flow. Differentiable search methods like DARTS tend to favor operations that facilitate gradient flow, such as skip connections (identity mappings).
In a shallow proxy, these skip connections look highly effective because they make optimization easy. However, as the search progresses, the network often suffers from topology collapse: the search algorithm fills the cells with skip connections, leaving no room for the complex convolutions needed for high representational capacity. When these "hollow" cells are stacked into a deep network, the final model lacks the depth of features required for complex tasks like ImageNet classification.
Progressive Growth as a Bridge
The most direct way to bridge this gap is to make the search environment look more like the evaluation environment. Lingxi Xie and colleagues introduced P-DARTS, which gradually increases the depth of the search network during the optimization process.
By starting shallow and ending deep, the algorithm "learns" to handle the gradient challenges of deeper structures. To keep the computational cost manageable, P-DARTS pairs this growth with search space pruning—dropping the least promising operations as the network gets deeper. This ensures that by the end of the search, the cell has been optimized for the actual conditions it will face at deployment.
Structural and Probabilistic Consistency
Other researchers have focused on making the search itself more robust to the depth gap without necessarily training massive networks.
- Structural Adjustments: Tao Huang and Yibo Yang proposed Stretchable Cells in Stretchable Cells Help DARTS Search Better. They explicitly model cell connectivity to ensure that the structural properties of the search supernet match the target architecture. Similarly, Shan You introduced EnTranNAS in Towards Improving the Consistency, Efficiency, and Flexibility of Differentiable Neural Architecture Search, which uses "Transit-cells" to mimic the final evaluation environment during search.
- Bayesian Regularization: Miao Zhang 0022 and Jilin Hu developed BaLeNAS (Differentiable Architecture Search via the Bayesian Learning Rule). Instead of treating architecture search as a simple point-optimization, they treat it as a distribution learning problem. This Bayesian approach naturally encourages exploration and prevents the search from prematurely collapsing into the "easy" but suboptimal solutions favored by shallow proxies.
- Efficiency Gains: To search on deeper networks directly, HyeongGwon Hong and Junmo Kim proposed EDAS (Efficient and Differentiable Architecture Search). By optimizing only one edge at a time, they reduce memory overhead, allowing researchers to search on deeper networks from the start, effectively bypassing the depth gap.
Why It Works
The common thread across these papers—from Bo Lyu's TND-NAS to Huafeng Qin's EM-DARTS—is the recognition that ranking consistency is more important than absolute accuracy during search.
If Architecture A is better than Architecture B at depth 8, but worse at depth 20, a shallow search is fundamentally deceptive. By bridging the depth gap—whether through progressive growth, structural mimicking, or Bayesian regularization—these methods ensure that the "winner" of the search phase remains the winner at deployment. This alignment transforms NAS from a fragile optimization trick into a reliable tool for discovering high-performance deep learning models.