Distance Preserving Neural Networks for Deterministic Uncertainty Estimation

This direction is split into other directions: bi-Lipschitz regularization. Its last version stays readable and citable here.

Version 1 of 128 papers43 conceptsfrozen 2026-10-04

Standard deep neural networks are notoriously overconfident. When presented with an image of a cat, they correctly predict "cat." But when presented with a random scribble or a medical scan they’ve never seen, they often predict "cat" with 99% confidence. This lack of "distance awareness" makes them dangerous in high-stakes deployments like autonomous driving or medicine.

To solve this, researchers traditionally use Deep Ensembles—training multiple models and checking if they disagree. While effective, ensembles are computationally expensive. A growing body of research, led by Joost van Amersfoort, Jishnu Mukhoti, and Andreas Kirsch, suggests we can achieve similar reliability in a single forward pass by preventing a phenomenon called feature collapse.

The Problem: Feature Collapse

In a standard neural network, the goal of the feature extractor is to map complex inputs (like pixels) into a latent space where they are easily separable by a linear classifier. However, as Joost van Amersfoort et al. identify in On Feature Collapse and Deep Kernel Learning for Single Forward Pass Uncertainty, this process is too efficient.

The network learns to "collapse" the entire manifold of training data into tight clusters. Crucially, it also maps out-of-distribution (OOD) data—things it has never seen—into these same clusters. Because the classifier only sees the collapsed features, it cannot distinguish between a "typical" training example and a completely novel input. The distance information from the original input space is lost.

The Solution: The Bi-Lipschitz Constraint

To fix this, we must ensure the network is "distance-aware." If two points are far apart in the input space, they should remain relatively far apart in the feature space. This is achieved by enforcing a bi-Lipschitz constraint on the network.

A function $f$ is bi-Lipschitz if there exist constants $L_1$ and $L_2$ such that for any inputs $x_1$ and $x_2$: $$L_1 ||x_1 - x_2|| \leq ||f(x_1) - f(x_2)|| \leq L_2 ||x_1 - x_2||$$

Intuitively, the $L_2$ (Lipschitz) bound prevents the network from being too sensitive to small noise (smoothness), while the $L_1$ bound prevents the network from collapsing distinct inputs into the same point (sensitivity). As noted in Deep Deterministic Uncertainty: A Simple Baseline, this constraint ensures that OOD data does not "hide" inside the training clusters in the latent space.

Implementing Distance Awareness

The most common way to enforce this constraint is through Spectral Normalization. By normalizing the weights of each layer by their largest singular value, researchers can bound the Lipschitz constant of the entire feature extractor.

  1. Regularized Feature Extraction: Apply spectral normalization to all layers. This prevents the "folding" of the input space that leads to feature collapse.
  2. Density Estimation: Once the feature space is regularized, we can use a simple density estimator—like a Gaussian Mixture Model or Gaussian Discriminant Analysis (GDA)—to model the distribution of the training data in that space.
  3. Uncertainty Calculation: At test time, if a new input's features fall into a low-density region (far from training clusters), the model flags high epistemic uncertainty (model ignorance).

This approach, termed Deep Deterministic Uncertainty (DDU) by Jishnu Mukhoti and Philip H. S. Torr in Deterministic Neural Networks with Inductive Biases Capture Epistemic and Aleatoric Uncertainty, allows a single network to distinguish between "confusing" data (aleatoric uncertainty) and "unseen" data (epistemic uncertainty).

Applications and Extensions

This principle of distance preservation is being applied across various domains:

By moving from "black box" feature extraction to distance-preserving representations, these methods provide a path toward deep learning models that "know what they don't know" without the computational burden of ensembles.

Papers in this version

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_distance_preserving_neural_networks_for__8e223a16,
  title = {Distance Preserving Neural Networks for Deterministic Uncertainty Estimation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/distance-preserving-neural-networks-for-deterministic-uncertainty-estimation/rdr_8e223a1694e049d89909df991e3a8107},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_distance_preserving_neural_networks_for__8e223a16_v1,
  title = {Distance Preserving Neural Networks for Deterministic Uncertainty Estimation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/distance-preserving-neural-networks-for-deterministic-uncertainty-estimation/rdr_8e223a1694e049d89909df991e3a8107/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}