In the standard mental model of deep generative models, the latent space is a flat, Euclidean plane. We pick two points, draw a straight line between them, and expect the decoder to produce a smooth, realistic transition. However, as Georgios Arvanitidis, Lars Kai Hansen, and Søren Hauberg argue in Latent Space Oddity: on the Curvature of Deep Generative Models, this "Euclidean assumption" is a fundamental misunderstanding of how these models actually work.
The reality is that the nonlinear decoder function warps the latent space into a complex, curved manifold. Treating this space as flat leads to "manifold mismatch," where the shortest path in the latent space (a straight line) does not correspond to the shortest path on the data manifold.
The Decoder as a Geometric Warper
When a Variational Autoencoder (VAE) or GAN learns a mapping from a low-dimensional latent space $\mathcal{Z}$ to a high-dimensional data space $\mathcal{X}$, it is essentially defining a surface. Because the decoder is a deep neural network with nonlinear activations, it stretches some regions of the latent space and compresses others.
Abhishek Kumar and P. Thomas Fletcher established a rigorous framework in The Riemannian Geometry of Deep Generative Models to treat these models as Riemannian manifolds. In this view, the "distance" between two points in latent space shouldn't be measured by a ruler on the latent grid, but by the "pullback metric" induced by the decoder. If a small step in latent space causes a massive change in the generated image, that region of the manifold is highly curved or "stretched."
Why Linear Interpolation Fails
The most visible symptom of ignoring this geometry is poor interpolation. In a flat space, the shortest path between two points is a straight line. But on a curved manifold, the shortest path is a geodesic.
As Francesco Ferroni and colleagues explain in Fast Approximate Geodesics for Deep Generative Models, linear paths often "fall off" the manifold. They pass through "dead zones"—regions of low data density where the model hasn't learned anything meaningful. This is why linear interpolations often look blurry or generate "nonsense" intermediate samples. By following the curvature of the manifold (the geodesic), we stay within the high-density regions where the data actually lives.
This isn't just a visual problem; it's a functional one. In PepCompass: Navigating peptide embedding spaces using Riemannian Geometry, Marcin Możejko and his team found that accounting for this curvature was the difference between success and failure in drug discovery. By navigating the manifold using Riemannian tools, they achieved a 100% validation rate for novel antimicrobial peptides, whereas Euclidean paths produced biologically invalid sequences.
Identifying the True Metric Structure
One of the persistent frustrations with deep latent variable models is their "unidentifiability": if you retrain the same model, the latent coordinates might end up in completely different places.
However, Stas Syrota and Yevgen Zainchkovskyy recently proved in Identifying Metric Structures of Deep Latent Variable Models that while the coordinates are arbitrary, the Riemannian metric is not. The distances and angles measured along the manifold are statistically identifiable. This means that if we stop looking at the raw coordinates and start looking at the manifold's geometry, we can extract reproducible, scientific conclusions from "black box" models.
Enforcing and Adapting to Geometry
Researchers are now moving beyond just analyzing curvature to controlling it. Two main strategies have emerged:
- Enforcing Flatness: In Enforcing Latent Euclidean Geometry in Single-Cell VAEs for Manifold Interpolation, Alessandro Palma and Fabian J. Theis introduced FlatVI. This framework regularizes the VAE during training to force the latent space to actually be flat. By aligning the model's internal geometry with Euclidean space, they make simple linear interpolation valid for predicting complex biological cellular trajectories.
- Geometry-Aware Priors: Instead of forcing the model to be flat, we can change the "map" it uses. Dimitrios Kalatzis proposed Variational Autoencoders with Riemannian Brownian Motion Priors, which replaces the standard Gaussian prior with a process that "walks" along the learned manifold. This ensures that the model's prior belief about the data distribution matches the actual warped geometry of the latent space.
Beyond Images: Networks and Motion
This geometric perspective is expanding into other domains where data is inherently non-Euclidean. Shane Lubold and Arun G. Chandrasekhar have shown that social networks often reside in hyperbolic or spherical latent spaces rather than Euclidean ones. Similarly, in robotics, Yonghyeon Lee uses Riemannian metrics in Isometric Motion Manifold Primitives to ensure that robot motion planning stays "isometric"—meaning distances in the latent space accurately reflect the physical similarity of different movements.
By treating the latent space as a Riemannian manifold, researchers are finally beginning to "see" the true shape of the data that deep learning models have been trying to describe all along.