Imagine you want to travel from New York to Tokyo. If you draw a straight line between them on a flat map, or worse, drill a straight tunnel directly through the center of the Earth, you are taking a shortcut through a space where humans cannot live. To measure the actual journey, you must follow the curved surface of the globe.
This is the core intuition behind manifold learning. In high-dimensional data science—whether dealing with pixel arrays, text embeddings, or neural network representations—data rarely fills the space uniformly. Instead, it concentrates on low-dimensional, highly curved surfaces called manifolds.
When we use standard linear metrics like Euclidean distance or cosine similarity to compare these data points, we make the mistake of drilling through the Earth. We calculate a straight-line "short-circuit" through empty space, ignoring the underlying geometry of the data.
Standard Euclidean Distance (Short-circuit through empty space)
●------------------------●
/ \
/ \
● ●
\ /
\ /
●------------------------●
Geodesic Distance (True path along the data manifold)
The Manifold-Metric Mismatch
The fundamental problem with flat metrics is that they assume the representation space is locally and globally linear. When data lies on a highly non-linear manifold, this assumption creates a severe geometric mismatch.
In a foundational single-author paper, Joshua B. Tenenbaum exposed this flaw by introducing Isomap in Mapping a Manifold of Perceptual Observations. If you have a sequence of images showing a face rotating, the images trace a curved path in pixel space. Two images of the face facing opposite directions might appear close in raw Euclidean distance if the lighting and background match, yet they are semantically far apart along the true manifold of physical rotation. Measuring straight-line distance through empty space ignores this geometry, leading to inaccurate similarity assessments.
This limitation is not confined to raw pixels; it persists inside modern machine learning architectures. For example, conventional multi-modal models like CLIP rely on the assumption that text and image embeddings are distributed uniformly on a high-dimensional sphere, using linear metrics like cosine similarity to determine semantic proximity. However, Mei et al. demonstrate that real-world image and text data actually reside on complex, non-linear manifolds. By replacing cosine similarity with geodesic distance—the shortest path along the data manifold—the authors significantly improved the ability of multi-modal models to distinguish between samples that appear superficially similar but are semantically distinct.
Similarly, Modell et al. proved that Large Language Models represent continuous concepts (such as time or color) as geometric manifolds. While cosine similarity works well for finding immediate, local neighbors, it becomes highly unreliable when comparing concepts that sit further apart in the underlying conceptual space.
Why Local is Linear, but Global is Curved
If global Euclidean distance fails, how do we calculate the true geodesic distance along an unknown, invisible manifold?
The solution relies on a simple principle of differential geometry: curved spaces look flat when you zoom in. Just as the Earth looks flat from a walking path, a highly curved data manifold is approximately linear within a sufficiently small neighborhood.
To exploit this, algorithms construct a nearest-neighbor graph. If two data points are very close, their ambient Euclidean distance is a reliable proxy for their manifold distance. By connecting every point only to its closest neighbors, we map out the local infrastructure of the manifold. To find the true distance between two distant points, we no longer draw a straight line; instead, we find the shortest path by hopping from neighbor to neighbor along the graph.
●------●------●
/ \
/ \
● ● <- Geodesic path chained via local linear steps
\ /
\ /
●------●------●
This graph-based chaining prevents the metric from "short-circuiting." As Balakrishnan et al. explain, using flat Euclidean distances across high-curvature manifolds causes unrelated regions of the data to incorrectly merge. Chaining local steps ensures that distinct folds of a data structure remain separate.
This insight underpins many modern architectures. For instance, in few-shot learning, where models must generalize from only a handful of examples, standard Euclidean metrics fail to capture the intricate structures of high-dimensional data. The transductive propagation network introduced by Liu et al. bypasses this by building a task-specific graph, propagating labels along the manifold structure rather than relying on flat distance calculations.
The Cost of Ignoring Geometry
When models ignore manifold structure, the errors ripple through downstream tasks like clustering, domain adaptation, and interpretability.
1. Broken Clustering and Anomaly Detection
Traditional clustering algorithms like K-Means assume that clusters form neat, spherical "blobs" in Euclidean space. But when data resides on complex, winding paths, points that are physically close in high-dimensional space may belong to entirely different semantic structures.
In unsupervised matrix factorization, the authors of Clustering by Nonnegative Matrix Factorization Using Graph Random Walk show that standard factorization techniques fail because they rely on long-range Euclidean distances. By replacing raw similarity matrices with smoothed versions derived from graph random walks, they bridge the gap and capture global connectivity. In anomaly detection, Manqi Zhao demonstrated that leveraging the geometric properties of nearest-neighbor graphs rather than standard Euclidean metrics prevents false alarms caused by the complex, low-dimensional layout of normal data.
2. Misaligned Domain Adaptation
In Unsupervised Domain Adaptation (UDA), we try to align a source dataset (e.g., synthetic images) with a target dataset (e.g., real photos). A popular tool for this is Optimal Transport (OT), which calculates the most efficient way to "move" one distribution to match another.
However, Maman et al. reveal that when transport algorithms rely on straight-line Euclidean distances, they routinely produce incorrect alignments. If data points from different semantic clusters happen to sit close to each other in the ambient space, standard OT will mistakenly map them together. By designing a geometric optimal transport framework that respects the underlying data path, the authors prevent these unrelated features from crossing over.
Euclidean Optimal Transport (Incorrect Alignment)
Source Manifold: [Class A] ----> (Short-circuit) ----> [Class B] Target Manifold
Geometric Optimal Transport (Correct Alignment)
Source Manifold: [Class A] --------(Follows Path)--------> [Class A] Target Manifold
3. Illusionary Model Interpretability
We often use distance metrics to explain how a model's internal logic changes during training or fine-tuning. If a model's embeddings shift, we calculate the Euclidean distance between the old and new representations.
But as Kondapaneni et al. point out, this assumes local linearity. If the underlying representation manifold is highly non-linear, these distance-based metrics produce misleading results that fail to reflect the actual semantic changes in the model. Similarly, Benfenati et al. reframe deep neural networks through the lens of singular Riemannian geometry, proving that traditional Euclidean analysis fails to capture how networks compress and partition information along low-dimensional manifolds.
Broader Geometric Frameworks
Recognizing the limitations of flat space has led researchers to explore alternative geometries entirely. When data has a specific structural constraint, flat rulers must be abandoned for specialized coordinate systems:
- Projective Geometry: Bi et al. introduced the Cayley-Klein metric framework to computer vision, proving that standard flat metrics like the Mahalanobis distance are merely special, restricted cases of a more general, curved geometric framework.
- Physical Systems: In complex engineering simulations like fluid dynamics, Dunhui Xiao demonstrated that standard linear reduction methods miss critical physics. By implementing a probabilistic manifold decomposition that utilizes geodesic distances, the system preserves the underlying physical constraints of the state space.
Ultimately, whether analyzing the representations of a large language model, aligning distributions across domains, or clustering complex images, respecting the intrinsic geometry of the data manifold is essential. Replacing rigid, straight-line Euclidean metrics with flexible, manifold-aware paths allows models to stop cutting through empty space and start navigating data as it truly exists.
Key Foundational Work
Mapping a Manifold of Perceptual Observations (1997) — contributed 1 concept elements to this direction.
Extended Isomap for Pattern Classification (2002) — contributed 1 concept elements to this direction.