Differentiable Discrete Selection for Neural Networks

This direction is merged into another direction: differentiable discrete sampling (differentiable categorical sampling). Its last version stays readable and citable here.

Version 1 of 162 papers62 conceptsfrozen 2026-10-04

Neural networks are fundamentally continuous machines. They thrive on smooth landscapes where small changes in input lead to small changes in output, allowing gradients to flow backward and guide learning. However, the real world—and efficient computation—is often discrete. We want models that can "skip" a word, "pick" a specific neighbor in a graph, or "decide" which sensor to activate.

Traditionally, these discrete choices (like an if/else statement or a top-k selection) are the enemies of backpropagation because their derivatives are either zero or undefined. This research direction focuses on Differentiable Discrete Selection: a suite of techniques that allow models to make hard, categorical decisions while remaining end-to-end trainable.

The Discreteness Dilemma

Consider the task of "speed reading." In Neural Speed Reading via Skim-RNN, Minjoon Seo and colleagues wanted a model that could choose to "skim" unimportant tokens using a tiny RNN. If the model makes a hard binary choice—either process or skip—the gradient cannot tell the model how to improve that choice. If you skip a word and the error goes up, the "skip" function doesn't have a slope to tell you how much less you should have skipped.

To solve this, researchers have moved away from fixed heuristics toward learnable selection policies. The goal is to treat the selection itself as a parameter to be optimized alongside the main task.

The Gumbel-Softmax Bridge

The most prevalent tool in this area is the Gumbel-Softmax trick (or concrete relaxation). It provides a way to sample from a categorical distribution in a way that gradients can pass through.

Instead of just picking the highest-scoring option (the argmax), the process is broken into two steps:

  1. Perturbation: Add random noise (from a Gumbel distribution) to the model's scores.
  2. Softening: Apply a softmax with a "temperature" parameter.

When the temperature is high, the output is a smooth, fuzzy distribution. As the temperature cools, the output approaches a "one-hot" vector—a hard discrete choice. This allows models like AdaViT by Lingchen Meng to learn which image patches or transformer heads to prune on-the-fly. By using this trick, the model can explore different pruning strategies during training and converge on a hard, efficient policy for inference.

Similar reparameterization logic is applied in GRAPES by Taraneh Younesian to sample graph neighbors and in FaMUS by Linchao Zhu to select which layers should receive gradient updates during meta-learning.

Beyond Gumbel: Perturbations and Relaxations

While Gumbel-Softmax is popular, it can be high-variance. Other researchers use different "relaxations" to bridge the gap:

  • Perturbed Maximums: In Differentiable Patch Selection for Image Recognition, Aravindh Mahendran and Jean-Baptiste Cordonnier add Gaussian noise to scores and average multiple solutions to create a smooth approximation of a top-k selection.
  • Differentiable RANSAC: Eric Brachmann and his team developed DSAC, which treats the selection of geometric hypotheses as a stochastic process. By calculating the expected loss over many samples, they make the notoriously non-differentiable RANSAC algorithm trainable.
  • Structured Sparsity: For tasks like fact-checking, Xuming Hu uses SCALE, which employs regularized linear programming (LP-MAP) to select contiguous sentences as evidence. This provides a deterministic and more stable gradient than sampling-based methods.

Efficiency through Adaptive Computation

A major driver for this research is the "Performance-Overhead Tradeoff." Modern models are often over-parameterized for "easy" inputs. Differentiable selection allows for Adaptive Inference:

  • Layer Skipping: Models like ConvNet-AIG and SkipRec learn to bypass entire neural layers for simple inputs, saving up to 30-50% of computation without losing accuracy.
  • Dynamic Sampling: In LighTN, Xu Wang uses a differentiable sampler to downsample massive point clouds, ensuring only the most task-relevant points are processed by the heavy transformer backbone.
  • Sensor Selection: For battery-constrained devices, Thomas Strypsteen and Alexander Bertrand use these mechanisms to dynamically turn off unnecessary sensors based on the current signal.

Sparsity as a Path to Interpretability

Finally, differentiable selection serves as a powerful tool for "Explainable AI." By forcing a model to select a sparse subset of features or events, we can see why it made a decision.

In Explainable History Distillation, Sishun Liu and Ke Deng use a differentiable binary mask to identify the smallest subset of past events that explain a future prediction. Because the selection is trained end-to-end, the "rationale" the model provides is not just a post-hoc guess but the actual data the model used to reach its conclusion. This same principle is used in Interventional Rationalization to ensure that selected text snippets have a true causal relationship with the model's output.

Papers in this version

Showing 60 of 62 papers.

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_differentiable_discrete_selection_for_ne_38fc03c4,
  title = {Differentiable Discrete Selection for Neural Networks},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/differentiable-discrete-selection-for-neural-networks/rdr_38fc03c45c254f2a9eb11e269eb9140d},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_differentiable_discrete_selection_for_ne_38fc03c4_v1,
  title = {Differentiable Discrete Selection for Neural Networks},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/differentiable-discrete-selection-for-neural-networks/rdr_38fc03c45c254f2a9eb11e269eb9140d/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}