Differentiable Architecture Search (DARTS), introduced by Liu et al. in DARTS: Differentiable Architecture Search, revolutionized how we design neural networks. Instead of treating architecture selection as a slow, discrete search over billions of possible graph topologies, DARTS relaxes the problem. It creates a dense supernet where every edge contains a weighted mixture of all possible operations (e.g., $3\times3$ convolutions, skip connections, max pooling). The weights of these mixtures are optimized using standard gradient descent.
Yet, this mathematical elegance hides a severe practical flaw known as the discretization gap (or discretization discrepancy). When the search phase ends, this fuzzy, continuous supernet must be pruned into a sparse, discrete subnet for final deployment—typically by keeping only the single operation with the highest weight on each edge.
During this transition, the model's performance frequently collapses. An architecture that achieved top-tier validation accuracy during the search phase suddenly underperforms when isolated and evaluated.
Why the Discretization Gap Occurs
To understand why this gap exists, we must look at how weights behave in a shared, continuous mixture versus a standalone structure.
Co-Adaptation vs. Isolation
In the continuous supernet, candidate operations do not run in isolation; they share the representational burden. As observed by Zhang et al. in Interpreting Operation Selection in Differentiable Architecture Search: A Perspective from Influence-Directed Explanations, the joint optimization of multiple operations behaves fundamentally differently from an isolated operation. An operation might receive a high architectural weight simply because it co-adapts effectively with neighboring parallel paths. When those parallel paths are abruptly pruned away, the selected operation is forced to perform alone, exposing its inability to handle the full data transformation by itself.
Sharp vs. Flat Minima
The geometry of the loss landscape during the search phase heavily influences how well an architecture survives pruning. In Understanding and Robustifying Differentiable Architecture Search, Zela demonstrated that continuous optimization tends to find "sharp" minima in the architecture parameter space. These solutions yield low validation loss when all paths are active and smoothly blended, but they are highly sensitive to structural perturbations. Snapping the continuous weights to a discrete structure acts as a massive perturbation, throwing the model out of the sharp minimum and degrading its performance.
Flawed Selection Heuristics
The standard DARTS protocol assumes that the magnitude of an architecture weight ($\alpha$) directly reflects the operation's standalone value. Cheng et al. challenged this assumption in Rethinking Architecture Selection in Differentiable NAS, showing that $\alpha$ values often reflect which operations optimize the fastest early in training (such as skip connections) rather than which operations are best in the final discrete model. This creates a profound disconnect between what the supernet optimizes and what the selection rule extracts.
Mechanisms for Closing the Gap
To prevent this performance collapse, researchers have developed several distinct strategies to align the continuous search phase with the discrete evaluation phase.
Continuous Supernet (Dense Mixture)
│
├─► 1. Progressive Annealing ──► Gradually forces weights to 0 or 1
├─► 2. Knowledge Distillation ─► Aligns soft proxy and hard subnet behaviors
├─► 3. Geometric Regularization ► Restricts search to naturally sparse spaces
│
▼
Discrete Subnet (Sparse Deployment)
1. Progressive Discretization and Annealing
Rather than maintaining a fully dense supernet until a sudden final pruning step, some methods introduce a gradual transition during training. In ASAP: Architecture Search, Anneal and Prune, the authors utilize a temperature-controlled annealing process to slowly fade out underperforming connections over time. This allows the remaining network weights to incrementally adapt to the absence of pruned paths.
Similarly, Tian et al. introduced Discretization-Aware Architecture Search, which applies an entropy-based regularization loss. This penalty pushes the continuous architectural weights toward binary values ($0$ or $1$) as the search progresses, ensuring that the supernet naturally mirrors a discrete architecture by the end of training.
2. Knowledge Distillation Across the Gap
Another strategy forces the discrete architecture to directly inherit the representational capacity of the continuous model via distillation. Li et al. demonstrated in Improving Differentiable Architecture Search via Self-Distillation that treating the supernet from previous iterations as a teacher flattens the loss landscape, ensuring that the optimal continuous parameters remain effective when the model is simplified.
This concept is also highly effective in model pruning. In S2HPruner: Soft-to-Hard Distillation Bridges the Discretization Gap in Pruning, Lin et al. introduced a decoupled bidirectional distillation paradigm. By co-training a soft continuous proxy network alongside its hard-pruned counterpart, the discrete model learns to mimic the higher representational capacity of the continuous space, eliminating the need for expensive post-pruning retraining phases.
3. Modifying Optimization Geometry and Constraints
Instead of fixing the gap after optimization, some researchers alter the optimization space itself to prevent dense co-adaptation from occurring. Li et al. proposed Geometry-Aware Gradient Algorithms for Neural Architecture Search, which replaces standard gradient descent with Mirror Descent on a simplex. This geometric constraint naturally biases the optimization toward sparse, discrete-like boundaries.
To eliminate the unfair advantages that certain operations gain during softmax competition, Chu et al. developed Fair DARTS: Eliminating Unfair Advantages in Differentiable Architecture Search. By replacing the softmax function with a sigmoid function, they allowed operations to develop independently rather than in a zero-sum competition, combining this with a specialized zero-one loss to minimize the discretization discrepancy.
Key Foundational Work
Differentiable Neural Architecture Search via Proximal Iterations (2019) — contributed 3 concept elements to this direction.
Fair DARTS: Eliminating Unfair Advantages in Differentiable Architecture Search (2019) — contributed 2 concept elements to this direction.
DATA: Differentiable ArchiTecture Approximation (2019) — contributed 2 concept elements to this direction.