Minimizing Discretization Discrepancy in Differentiable Architecture Search

This direction is merged into another direction: discretization gap. Its last version stays readable and citable here.

Version 1 of 112 papers18 conceptsfrozen 2026-10-04

Differentiable Neural Architecture Search (DNAS) transformed the field of automated model design by turning a discrete combinatorial problem into a continuous optimization one. In the popular DARTS framework, instead of picking one operation (like a $3 \times 3$ convolution) for a network edge, we maintain a weighted mixture of all possible operations. We then use gradient descent to learn these weights.

However, a fundamental tension exists: we train a "supernet" that is a fuzzy mixture of operations, but we must eventually "discretize" it by throwing away all but the strongest operation to deploy a final model. This transition often leads to a discretization discrepancy—a performance collapse where the final discrete model behaves nothing like the continuous supernet it was derived from.

The Problem of the Fuzzy Supernet

In standard DNAS, the architectural weights are typically passed through a softmax function. This creates a competition where weights sum to one, but it doesn't necessarily force the weights to become binary (0 or 1). As Konstanty Subbotko, Wojciech Jablonski, and Piotr Biliński argue in The devil is in discretization discrepancy, this "fuzziness" allows the supernet to cheat. It might achieve high accuracy by co-adapting multiple weak operations, a luxury the final single-path model won't have.

This often manifests as a "skip-connection collapse." Because identity mappings are easier to optimize early in training, the supernet over-relies on them. When the search ends and we prune other operations, the resulting model is shallow and underperforms.

Forcing Binary Decisions via Entropy

To bridge this gap, researchers have moved toward forcing the optimizer to make hard choices during the search phase itself. The intuition is simple: if the architectural weights are pushed toward 0 or 1, the supernet becomes a closer approximation of the final discrete model.

One of the most direct ways to achieve this is through entropy regularization. By adding a penalty to the loss function that is minimized when entropy is zero, the search process is incentivized to "sharpen" its distribution. Yuan Gao and colleagues in MTL-NAS use this to create task-agnostic models that can be evaluated immediately after search without the traditional, expensive retraining phase. Similarly, Egor Shvetsov and Alexey Zaytsev apply entropy-based losses in QuantNAS to find architectures that are naturally robust to the noise of low-bit quantization.

Geometry and Loss Landscapes

Beyond simple penalties, the geometry of the optimization itself can be redesigned to favor sparsity. Liam Li, Mikhail Khodak, and their team proposed the Geometry-Aware Exponentiated Algorithm (GAEA) in Geometry-Aware Gradient Algorithms for Neural Architecture Search. By replacing standard SGD with Mirror Descent—an optimizer that respects the simplex geometry of architectural weights—they achieve rapid entropy reduction. This "geometry-aware" approach ensures the supernet converges to a sparse configuration naturally, minimizing the gap between the continuous search and the discrete result.

Another influential approach, Fair DARTS, by Xiangxiang Chu and Bo Zhang, identifies that the softmax function itself is part of the problem because it enforces an "unfair" competition where operations cannot grow independently. In Fair DARTS: Eliminating Unfair Advantages in Differentiable Architecture Search, they replace softmax with a sigmoid function and introduce a zero-one loss. This loss function is shaped like a "W," with minima at 0 and 1 and a maximum at 0.5, explicitly pushing every architectural weight toward a binary state.

The ultimate goal of these techniques is to move away from the rigid, two-stage "search then retrain" pipeline. If the discretization discrepancy is low enough, the weights learned during the search can be used directly.

In FX-DARTS, Xuan Rao and Cesare Alippi introduce Entropy-based Super-network Shrinking (ESS). This allows the network to settle into a discrete state even in "unconstrained" search spaces where the number of inputs and the topology aren't fixed by human designers. By the time the search concludes, the supernet has effectively "shrunk" into the final architecture.

Similarly, Yunjie Tian and Lingxi Xie show in IF-NAS that regularization toward binary values is essential for scaling DNAS to complex macro-architectures. Without these mechanisms, the interference between overlapping skip-connections (interleaving) makes the search space too noisy to navigate.

By treating the discretization gap not as an inevitable byproduct of the search, but as an optimization objective to be minimized, this line of research has made NAS more stable, more efficient, and significantly more reliable for real-world deployment.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_minimizing_discretization_discrepancy_in_908b3f89,
  title = {Minimizing Discretization Discrepancy in Differentiable Architecture Search},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/minimizing-discretization-discrepancy-in-differentiable-architecture-search/rdr_908b3f8928ea42bf94ae76f802544a89},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_minimizing_discretization_discrepancy_in_908b3f89_v1,
  title = {Minimizing Discretization Discrepancy in Differentiable Architecture Search},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/minimizing-discretization-discrepancy-in-differentiable-architecture-search/rdr_908b3f8928ea42bf94ae76f802544a89/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}