In traditional machine learning, we represent entities—whether they are words, images, or physical states—as "points" in a high-dimensional vector space. While efficient, this deterministic approach assumes we are perfectly certain about every feature.
Recent research across computer vision, natural language processing, and control theory suggests a more robust alternative: representing data as Gaussian distributions. By replacing a single vector with a mean ($\mu$) and a covariance matrix ($\Sigma$), models can explicitly represent ambiguity, noise, and the relationships between features.
The Overconfidence of Point Embeddings
A standard embedding maps an input to a fixed coordinate. However, in tasks like Gaussian Embeddings for Collaborative Filtering, Patrick Gallinari notes that this fails to capture "cold start" scenarios. If a user has only watched one movie, our "point" estimate of their preference is likely wrong. By using a distribution, the model can express high variance (uncertainty) for new users and low variance for long-time users.
Similarly, in Probabilistic Language-Image Pre-Training, Sanghyuk Chun and Wonjae Kim argue that a single caption can describe many different images. A point embedding forces a one-to-one mapping that doesn't exist in reality. A Gaussian representation allows the model to encompass a "region" of semantic meaning.
The Diagonal Covariance Shortcut
The primary challenge in probabilistic modeling is the covariance matrix $\Sigma$. For a $D$-dimensional embedding:
- A Full Covariance matrix has $D^2$ parameters, capturing how every feature correlates with every other feature.
- A Diagonal Covariance matrix has only $D$ parameters, assuming every feature is independent.
To maintain computational tractability, most researchers—from Avihai Mejer in Confidence in Structured-Prediction Using Confidence-Weighted Models to Aleksandar Bojchevski in Deep Gaussian Embedding of Graphs—rely on the diagonal assumption. This reduces the math from cubic complexity to linear, making it possible to scale to millions of parameters.
Why Independence is a Dangerous Assumption
While diagonal matrices are fast, they are "axis-aligned." They can model how much noise is in a specific feature, but they cannot model how features move together.
David J. Montana demonstrated in A Weighted Probabilistic Neural Network that diagonal models fail to account for rotations in the input space. If your data is tilted at a 45-degree angle relative to your axes, a diagonal matrix must over-expand to cover the data, introducing "empty" uncertainty that doesn't actually exist.
In robotics, this has physical consequences. Huseyin Coskun and Federico Tombari point out in Long Short-Term Memory Kalman Filters that the position of a human elbow is highly correlated with the shoulder. A diagonal model treats these joints as independent, leading to jittery, anatomically impossible pose estimates.
Capturing Complex Dependencies
To move beyond the diagonal limitation without the cost of a full matrix, researchers use several strategies:
- Learned Priors: Rajat Raina and Andrew Y. Ng showed in Constructing informative priors using transfer learning that we can learn correlations from a related "source" task and bake them into the covariance of a "target" task.
- Covariance Tying: In speech synthesis, Heiga Zen and Keiichiro Oura utilized Covariance-Tying to share a single full covariance matrix across multiple states, capturing global correlations while keeping the parameter count low.
- Moment Matching: For continuous systems, Andreas Look and Jan Peters developed Bidimensional Moment Matching, which approximates how uncertainty propagates through a neural network without needing expensive sampling.
The Future of Probabilistic Latents
The shift toward Gaussian representations is ultimately about calibration. As Lars Rosenbaum argues in Labels Are Not Perfect, even our "ground truth" labels are often uncertain. By treating every layer of the stack—from input to latent space to output—as a distribution, we create models that don't just predict, but also know when they are likely to be wrong.
The ongoing challenge, as highlighted by Sanghyuk Chun in Improved Probabilistic Image-Text Representations, is finding the "sweet spot" where we capture enough correlation to be accurate without incurring the memory overhead that makes full-covariance models prohibitive for real-time use.