Gaussian Representations for High Dimensional Uncertainty

This direction is split into other directions: diagonal covariance assumption. Its last version stays readable and citable here.

Version 1 of 1265 papers280 conceptsfrozen 2026-10-04

In traditional machine learning, we represent entities—whether they are words, images, or physical states—as "points" in a high-dimensional vector space. While efficient, this deterministic approach assumes we are perfectly certain about every feature.

Recent research across computer vision, natural language processing, and control theory suggests a more robust alternative: representing data as Gaussian distributions. By replacing a single vector with a mean ($\mu$) and a covariance matrix ($\Sigma$), models can explicitly represent ambiguity, noise, and the relationships between features.

The Overconfidence of Point Embeddings

A standard embedding maps an input to a fixed coordinate. However, in tasks like Gaussian Embeddings for Collaborative Filtering, Patrick Gallinari notes that this fails to capture "cold start" scenarios. If a user has only watched one movie, our "point" estimate of their preference is likely wrong. By using a distribution, the model can express high variance (uncertainty) for new users and low variance for long-time users.

Similarly, in Probabilistic Language-Image Pre-Training, Sanghyuk Chun and Wonjae Kim argue that a single caption can describe many different images. A point embedding forces a one-to-one mapping that doesn't exist in reality. A Gaussian representation allows the model to encompass a "region" of semantic meaning.

The Diagonal Covariance Shortcut

The primary challenge in probabilistic modeling is the covariance matrix $\Sigma$. For a $D$-dimensional embedding:

  • A Full Covariance matrix has $D^2$ parameters, capturing how every feature correlates with every other feature.
  • A Diagonal Covariance matrix has only $D$ parameters, assuming every feature is independent.

To maintain computational tractability, most researchers—from Avihai Mejer in Confidence in Structured-Prediction Using Confidence-Weighted Models to Aleksandar Bojchevski in Deep Gaussian Embedding of Graphs—rely on the diagonal assumption. This reduces the math from cubic complexity to linear, making it possible to scale to millions of parameters.

Why Independence is a Dangerous Assumption

While diagonal matrices are fast, they are "axis-aligned." They can model how much noise is in a specific feature, but they cannot model how features move together.

David J. Montana demonstrated in A Weighted Probabilistic Neural Network that diagonal models fail to account for rotations in the input space. If your data is tilted at a 45-degree angle relative to your axes, a diagonal matrix must over-expand to cover the data, introducing "empty" uncertainty that doesn't actually exist.

In robotics, this has physical consequences. Huseyin Coskun and Federico Tombari point out in Long Short-Term Memory Kalman Filters that the position of a human elbow is highly correlated with the shoulder. A diagonal model treats these joints as independent, leading to jittery, anatomically impossible pose estimates.

Capturing Complex Dependencies

To move beyond the diagonal limitation without the cost of a full matrix, researchers use several strategies:

  1. Learned Priors: Rajat Raina and Andrew Y. Ng showed in Constructing informative priors using transfer learning that we can learn correlations from a related "source" task and bake them into the covariance of a "target" task.
  2. Covariance Tying: In speech synthesis, Heiga Zen and Keiichiro Oura utilized Covariance-Tying to share a single full covariance matrix across multiple states, capturing global correlations while keeping the parameter count low.
  3. Moment Matching: For continuous systems, Andreas Look and Jan Peters developed Bidimensional Moment Matching, which approximates how uncertainty propagates through a neural network without needing expensive sampling.

The Future of Probabilistic Latents

The shift toward Gaussian representations is ultimately about calibration. As Lars Rosenbaum argues in Labels Are Not Perfect, even our "ground truth" labels are often uncertain. By treating every layer of the stack—from input to latent space to output—as a distribution, we create models that don't just predict, but also know when they are likely to be wrong.

The ongoing challenge, as highlighted by Sanghyuk Chun in Improved Probabilistic Image-Text Representations, is finding the "sweet spot" where we capture enough correlation to be accurate without incurring the memory overhead that makes full-covariance models prohibitive for real-time use.

Papers in this version

Showing 60 of 265 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_gaussian_representations_for_high_dimens_a30881bd,
  title = {Gaussian Representations for High Dimensional Uncertainty},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/gaussian-representations-for-high-dimensional-uncertainty/rdr_a30881bdc3c747229658f8263ea66356},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_gaussian_representations_for_high_dimens_a30881bd_v1,
  title = {Gaussian Representations for High Dimensional Uncertainty},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/gaussian-representations-for-high-dimensional-uncertainty/rdr_a30881bdc3c747229658f8263ea66356/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}