When we apply manifold learning algorithms like Locally Linear Embedding (LLE), Laplacian Eigenmaps, or UMAP, we expect them to unroll and capture the true intrinsic dimensionality of our data. However, these algorithms suffer from a systemic vulnerability: when a data manifold is "long and skinny"—possessing a large aspect ratio—the algorithms frequently undergo manifold collapse, squashing multi-dimensional surfaces into lower-dimensional representations (such as flattening a 2D ribbon into a 1D line).
To understand why this happens, we have to look at the mathematical tension between two core components of dimensionality reduction: the local geometric objective and the global normalization constraints.
The Intuition: The Long Ribbon Problem
Imagine a dataset distributed along a long, narrow 2D ribbon wrapped in a high-dimensional space. The ribbon has two intrinsic coordinates: its length (the dominant direction) and its width (the minor direction).
Intuitively, an embedding algorithm should assign its first coordinate to the length and its second coordinate to the width. However, because the length is orders of magnitude larger than the width, standard algorithms often choose a second coordinate that represents a harmonic (a wave-like function) of the length axis rather than capturing the width axis. As a result, both embedding dimensions map the long axis, and the width dimension is completely lost.
As explored by Jesse He et al. in Product Manifold Learning with Independent Coordinate Selection, when a manifold exhibits a large aspect ratio, the dominant eigenvectors of the graph Laplacian capture variations exclusively along the longest dimension. The shorter, minor dimensions are ignored or collapsed entirely, leading to an irreversible loss of structural information.
The Mechanism: The Price of Normalization
Why do algorithms actively prefer redundant harmonics over independent dimensions? The answer lies in what Yair Goldberg et al. term "the price of normalization" in their foundational work, Manifold Learning: The Price of Normalization.
To prevent trivial solutions where all data points collapse into a single point, normalized-output algorithms (such as LLE, Laplacian Eigenmaps, and Diffusion Maps) enforce quadratic normalization constraints. These constraints mandate that the embedding coordinates must maintain a specific variance and remain orthogonal to one another.
When the underlying data manifold is highly stretched, a conflict arises:
- The Cost Function penalizes local distortions. It wants to map nearby points in the original space to nearby points in the embedding space.
- The Normalization Constraint forces the second embedding dimension to have the same total variance as the first.
If the algorithm chooses the narrow width as the second dimension, it must drastically stretch it to satisfy the variance requirement. This stretching incurs a massive penalty in the cost function. Alternatively, the algorithm can choose a higher-order harmonic of the long axis. Because the long axis already has massive variance, its harmonics can satisfy the variance constraint with a much lower energy penalty.
As demonstrated by Yair Goldberg et al. in their earlier analysis, once the aspect ratio of a manifold exceeds a specific mathematical threshold, the optimization framework mathematically favors collapsing the multi-dimensional surface into a lower-dimensional projection to minimize the total energy.
True Manifold (High Aspect Ratio)
[==================================================] <- Width is small but independent
Collapsed Embedding (Redundant Harmonics)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ <- Width is lost; 2nd dimension is just a wave along the length
The Redundancy Trap and Rank Drop
This collapse manifests as a drop in the rank of the mapping. In Selecting the independent coordinates of manifolds with large aspect ratios, Marina Meila et al. show that the primary failure mode on high-aspect-ratio data is this exact rank deficiency. The selected coordinates redundantly track dominant geodesic directions while completely neglecting independent dimensions.
Compounding this issue, standard data preprocessing steps often exacerbate the problem. The authors note that common data normalization procedures can inadvertently increase the aspect ratio of a manifold, making the data even more "long and skinny" before the embedding algorithm even begins.
This phenomenon is not isolated to classical spectral methods. In modern visualization tools like $t$-SNE and UMAP, normalization constants play a similarly dominant role. Sebastian Damrich et al. demonstrate in From t-SNE to UMAP with contrastive learning that the visual compactness of clusters is heavily driven by implicit normalization constants within their loss functions. A massive normalization constant distorts the embedding space, forcing points into tighter, potentially artificial arrangements.
Similarly, in the context of sparse subspace clustering, Mustafa Devrim Kaba et al. show in A Nullspace Property for Subspace-Preserving Recovery that certain geometric normalization techniques (like infinity-norm scaling) can inadvertently destroy the structural conditions required to preserve underlying subspaces.
Restoring Independence
To overcome normalization-induced distortion, researchers have developed techniques designed to filter out redundant harmonics and force the selection of truly independent coordinates.
In Minimally Redundant Laplacian Eigenmaps, David Pfau et al. introduce a modification (MR-LEM) that filters eigenvectors based on "unpredictability." If a higher-order eigenvector can be easily predicted from the already selected eigenvectors, it is discarded as a redundant harmonic. This forces the algorithm to bypass the dominant axis's harmonics and unearth the latent, independent dimensions of the manifold.
By shifting the focus from pure variance maximization to coordinate independence, these modern approaches prevent dimensional collapse, allowing algorithms to faithfully preserve the full geometric dimensionality of highly anisotropic data.
Key Foundational Work
Manifold Learning: The Price of Normalization (2008) — contributed 8 concept elements to this direction.
Manifold Learning: The Price of Normalization (2008) — contributed 6 concept elements to this direction.
Minimally Redundant Laplacian Eigenmaps (2018) — contributed 1 concept elements to this direction.