Bridging the Discretization Gap in Differentiable Search

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2147 papers209 conceptsfrozen 2026-10-04

In the quest to automate neural network design, Differentiable Architecture Search (DARTS) emerged as a breakthrough by turning a discrete combinatorial problem—choosing which operations to place in a network—into a continuous optimization problem. Instead of picking one operation (like a $3\times3$ convolution) and discarding others, DARTS maintains a "supernet" where every edge is a weighted mixture of all possible operations.

However, this convenience comes with a hidden cost: the discretization gap. This gap is the performance drop that occurs when we transition from the "soft" weighted mixture used during training to the "hard" single-operation architecture used in deployment.

The Supernet Illusion

The core intuition behind the discretization gap is that a mixture of operations behaves fundamentally differently than any single operation in isolation. As Xiangxiang Chu and colleagues observe in Fair DARTS, the continuous supernet often develops an "unfair advantage" by co-adapting multiple operations. For example, a mixture of a weak convolution and a skip connection might perform well together, but if the search process eventually prunes the convolution, the remaining skip connection may be useless on its own.

This discrepancy often leads to a "performance collapse," where the search process favors operations that optimize quickly in a mixture—like identity mappings—rather than those that are best for the final, standalone model. Miao Zhang 0022 notes in Interpreting Operation Selection that the best-performing operation in a mixture is not necessarily the best when selected individually.

Enforcing Discreteness During Training

To bridge this gap, researchers have moved toward "discrete-aware" optimization. The goal is to make the training environment look as much like the deployment environment as possible.

One approach is to use proximal iterations. In NASP, quanming yao and others treat the architecture search as a constrained optimization problem. By enforcing discrete choices throughout the search phase, they ensure the weights are optimized for the exact structure that will be deployed, eliminating the sudden shock of pruning at the end.

Another strategy involves stochastic sampling. In Mind the Gap, Shakir Yousefi and team use Gumbel noise to perform discrete selections in the forward pass while maintaining differentiability in the backward pass. This ensures the model's internal dynamics are aligned with its eventual inference behavior, which they show can reduce the performance drop by up to 98% in logic gate networks.

Gradual Transitions and Annealing

Rather than jumping from a soft mixture to a hard choice in one step, many modern methods use annealing or scheduled transitions.

  • Scheduled Pruning: Zhaofan Qiu introduces SDAS, which gradually "fixes" operations during training. This allows the network weights to slowly adapt to the increasing sparsity of the architecture.
  • Indicator Search: In DAIS, Ning Liu uses an annealing process to transition from continuous importance scores to binary channel selections, preventing the accuracy loss typically seen when filters are physically removed.

Soft-to-Hard Distillation

A more recent evolution is the use of distillation to "teach" the discrete model how to mimic the continuous one. Weihao Lin proposes S2HPruner, which simultaneously trains a soft proxy network and a hard pruned counterpart. By using bidirectional distillation, the hard model learns to capture the representational capacity of the soft mixture, effectively "bridging" the gap through knowledge transfer rather than just architectural alignment.

The discretization gap is not unique to NAS; it is a fundamental challenge in any domain where continuous proxies are used for discrete decisions.

  • Combinatorial Optimization: Nikolaos Karalias shows in Geometric Algorithms that models not optimized for the specific rounding procedure used at inference will yield sub-optimal solutions.
  • Hardware Constraints: For IoT devices, Matteo Risso demonstrates in DUCCIO that using discrete sampling is essential for accurate hardware cost estimation. Without it, a model might appear to satisfy memory limits in its continuous form but fail catastrophically once discretized for the chip.

By treating discreteness as a first-class citizen during training—rather than an afterthought—these methods ensure that the "intelligence" discovered during search actually survives the transition to the real world.

Papers in this version

Showing 60 of 147 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 100 papers
  2. Version 1 · 2026-10-04 · 147 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_discretization_gap_cf62b5f8,
  title = {discretization gap},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/discretization-gap/rdr_cf62b5f8cb8146abb0deef64de69de43},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_bridging_the_discretization_gap_in_diffe_cf62b5f8_v1,
  title = {Bridging the Discretization Gap in Differentiable Search},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/bridging-the-discretization-gap-in-differentiable-search/rdr_cf62b5f8cb8146abb0deef64de69de43/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}