When we train a deep generative model—such as a Variational Autoencoder (VAE) or a Generative Adversarial Network (GAN)—we map a complex, high-dimensional data distribution (like images of faces or audio waveforms) into a lower-dimensional latent space. A long-standing promise of these models is semantic manipulation: if you want to smoothly transform a picture of a closed mouth into an open smile, you should be able to draw a straight line between their latent coordinates and decode the points along the way.
Yet, if you have ever tried this, you have likely encountered a frustrating phenomenon. Instead of a smooth, natural morph, the intermediate images often become blurry, exhibit "ghosting" artifacts, or completely lose the identity of the subject.
This happens because of a fundamental misunderstanding of latent space geometry. We routinely treat latent spaces as flat, Euclidean environments. In reality, they are highly curved, anisotropic, and non-convex manifolds.
The Illusion of Flatness
To understand why linear interpolation fails, we must look at the relationship between the low-dimensional latent space and the high-dimensional observation space.
While an image might consist of millions of pixels, the set of all valid images (e.g., realistic human faces) occupies a much smaller, highly curved sub-surface known as the data manifold. As early as 1994, Bregler et al. demonstrated that linear transitions between two valid points on an image manifold routinely cross through regions of invalid space, resulting in visually nonsensical overlays rather than physical transformations.
Linear Path (Euclidean Shortcut) -> Cuts through empty, low-density space (Blurry/Invalid)
●-----------------------------------------●
/ \
/ \
● ●
Start Point End Point
\ /
\ /
●-------------------●-----------------------●
Geodesic Path (Follows the Curved Data Manifold)
When we train a deep generative model, the generator or decoder function is a highly non-linear neural network. As Arvanitidis et al. and Michelis et al. argue, this non-linear mapping warps the latent space. Even if we force the latent coordinates to follow a simple, flat prior distribution (like a standard Gaussian), the underlying geometry becomes severely distorted.
When you draw a straight line in a flat latent space, that path does not map to a straight line in the data space. Instead, the path takes a geometric "shortcut" through the interior of the manifold, traversing regions where the model has seen little to no training data. As a result, the decoder encounters latent vectors in these "holes" or low-density regions that it does not know how to realistically reconstruct, leading to the familiar blurriness and artifacts highlighted by Chadebec et al. and Bhalodia et al..
This geometric mismatch manifests across various architectures and modalities:
- Identity Bleeding: In autoencoders, straight-line paths often cause an object's unique characteristics to morph into entirely different classes mid-trajectory rather than preserving its core identity, a limitation studied by Connor et al..
- Spherical Violations: In models that rely on normalized embeddings (like CLIP), performing linear interpolation pulls the path into the interior of the hypersphere, violating the unit-norm constraint and creating synthetic samples that break the model's architectural assumptions, as noted by Oh and Quamer et al..
- Physical Implausibility: For 3D shape generation, Cosmo et al. showed that unconstrained latent paths yield intermediate shapes that melt or crumple because the latent space lacks grounding in the object's intrinsic surface metrics.
Walking the Curve: Riemannian Geodesics
If straight lines take us off the manifold, how do we stay on it? The answer lies in Riemannian geometry. Instead of calculating the shortest path using Euclidean distance, we must find the geodesic—the shortest path that conforms to the curvature of the learned manifold.
To do this, researchers compute a "pull-back" Riemannian metric tensor using the Jacobians of the generator network. This metric acts as a local scaling factor: in regions of high uncertainty or low data density, the metric expands distances, making those regions computationally "expensive" to cross. Consequently, a geodesic path will actively bend around the empty holes in the data distribution, staying firmly within high-density, realistic regions.
However, calculating true Riemannian geodesics requires solving complex differential equations involving network Jacobians, which Arvanitidis et al. note are notoriously unstable and computationally expensive. To make manifold-aware interpolation practical, recent work focuses on clever approximations:
- Graph-Based Approximations: Ferroni et al. bypassed non-linear optimization entirely by constructing a graph over encoded data points and using an A* search algorithm. This allows the system to find smooth, manifold-adhering paths in milliseconds rather than minutes.
- Density Priors as Geometry: Rather than calculating expensive pull-back metrics from the decoder, Arvanitidis et al. proposed defining a Riemannian metric that is simply inversely proportional to a learnable prior density. Shortest paths are thus naturally magnetized toward high-density data clusters.
Straightening the Manifold
Rather than changing how we navigate the latent space after training, an alternative paradigm seeks to change how the model constructs the latent space during training. If we can force the network to learn a flat or isometric representation, linear interpolation will naturally work.
For instance, Braunsmann et al. introduced a regularization technique that penalizes bending and distortion by training on triplets of nearby points. This forces the latent space to become isometric to the data manifold. When the latent space preserves the intrinsic distances and angles of the original data, straight Euclidean paths suddenly align with natural geodesic paths.
Similarly, Heo developed an isometric regularization framework for functional data, ensuring that linear paths correspond to constant-velocity, physically meaningful transitions in the output space, eliminating artifacts like warping or flickering.
Why It Matters
Moving away from the Euclidean latent space assumption is not just a mathematical nicety; it fundamentally changes model behavior and reliability across domains:
- Explainable AI: Feature attribution methods like Integrated Gradients often assume a straight-line interpolation between a baseline and an input. As Thabtah et al. point out, if this path goes off-manifold into unrealistic distributions, the resulting model explanations can become highly misleading.
- Robustness and Generalization: Smith et al. demonstrated that models can achieve high statistical accuracy while failing to learn the true connectivity of the data. Constraining interpolations to the true manifold prevents models from generating adversarial or invalid states during generalization tasks.
- Data Augmentation: Techniques like Mixup rely on interpolating between training samples to create synthetic data. By shifting from Euclidean interpolation to non-Euclidean spaces—such as the hyperbolic Poincaré ball utilized by Thakkar et al.—models can capture complex hierarchical structures without distortion, significantly improving performance in low-resource settings.
By treating the internal representations of deep networks as curved, dynamic landscapes rather than flat sheets of paper, we unlock the ability to navigate them safely, generating transitions that are not just mathematical averages, but realistic paths along the data manifold.
Key Foundational Work
Manifold Preserving Hierarchical Topic Models for Quantization and Approximation (2013) — contributed 1 concept elements to this direction.
Nonlinear Image Interpolation using Manifold Learning (1994) — contributed 1 concept elements to this direction.