In the ideal world of generative modeling, a model trained on a specific dataset—say, photographs of faces—should act as a gatekeeper. If you show it a new face, it should assign it a high probability density (likelihood). If you show it a picture of a cat or random noise, it should assign it a low likelihood.
However, a series of influential papers has revealed a startling "likelihood paradox": deep generative models frequently assign higher likelihoods to simple, out-of-distribution (OOD) data than to the actual data they were trained on. As Eric Nalisnick and colleagues first detailed in Do Deep Generative Models Know What They Don't Know?, a model trained on the complex CIFAR-10 dataset will often find the simpler SVHN (digits) dataset more "probable" than its own training images.
The Gravity of Simplicity
Why does a model trained on faces prefer a blank wall or a simple geometric shape? The answer lies in complexity bias. Generative models are essentially compression engines; they try to find the most efficient way to represent data.
Genki Osada and colleagues describe this as Density Concentration Attraction for Simpleness (DCAS). In models with Gaussian latent spaces (like VAEs or Normalizing Flows), simple, low-entropy inputs require fewer features to describe. Consequently, the model's transformation maps these simple inputs toward the high-density center of the latent distribution (the origin). As Osada notes in Understanding Likelihood of Normalizing Flow and Image Complexity through the Lens of Out-of-Distribution Detection, these simple images act like "gravity wells," pulling probability mass toward them simply because they lack the structural variance that pushes complex images toward the periphery.
This bias is so strong that the "most likely" samples in a model are often not realistic at all. Rafal Karczewski demonstrates in Diffusion Models as Cartoonists! that the highest-density regions of diffusion models contain blurry, simplified "cartoons" rather than high-fidelity images. The model mathematically prefers these samples because they are easier to represent, even if they never appeared in the training set.
Density vs. Probability Mass
To understand why this happens without the model actually generating these simple anomalies, we must distinguish between density and probability mass.
Hamidreza Kamkari and co-authors provide a geometric explanation in A Geometric Explanation of the Likelihood OOD Detection Paradox. They argue that simple OOD data often sits on lower-dimensional manifolds. While the density at a specific point in these regions might be high, the total probability mass (the integral of density over a volume) is negligible. The model assigns high likelihood to a single "simple" pixel pattern, but because that pattern is so specific and low-dimensional, the model is unlikely to ever stumble upon it during random sampling.
Correcting the Bias
Researchers have proposed several ways to bypass this complexity trap for better OOD detection:
- Dual Priors: Xuming Ran proposed the Bigeminal Priors VAE (BPVAE), which uses two collaborative latent priors—one specifically for complex features and one for simple ones—to decouple semantic similarity from structural complexity.
- Typicality and Multiple Statistics: Instead of looking at a single likelihood value, Warren Richard Morningstar introduced DoSE (Density of States Estimator). This method checks if a sample is "typical" across multiple model statistics, rather than just "probable."
- Reconstruction Error: Wenyu Jiang and Pablo M. Olmos suggest that reconstruction error is a more robust signal. In READ, the authors show that while a simple OOD image might have high likelihood, it often has a distinct reconstruction signature when compared to in-distribution data of similar complexity.
- Algorithmic Information Theory: Lucas Theis proposes using a "universal critic" to detect "randomness deficiency." If an image is too easily described (too low-complexity) compared to what the training distribution expects, it is flagged as unrealistic.
By accounting for the inherent "simpleness" of an input, these methods transform generative models from biased density estimators into robust detectors of genuine semantic novelty.