Traditional dimensionality reduction techniques like Principal Component Analysis (PCA) operate on a global scale, seeking to preserve the directions of maximum variance across an entire dataset. However, high-dimensional data—such as images, text, or LiDAR point clouds—often resides on a lower-dimensional, non-linear surface known as a manifold. In these cases, global Euclidean distances are misleading; two points might appear close in high-dimensional space simply because the manifold is "folded," even if they are far apart along the manifold's surface.
To address this, a family of manifold-aware techniques focuses on locality preservation. The core intuition is simple: while global relationships may be distorted by the manifold's curvature, local relationships remain approximately linear and trustworthy. By ensuring that nearby points in the high-dimensional space remain neighbors in the low-dimensional embedding, these methods "unfold" the data without tearing its intrinsic structure.
The Principle of Local Isometry
The most direct way to preserve a manifold is to maintain local isometry—the distances and angles between a point and its immediate neighbors. Weinberger et al. and Saul proposed Maximum Variance Unfolding (MVU), which formalizes this as a convex optimization problem.
Instead of trying to model the global curvature, the authors suggest maximizing the total variance of the data (pulling the points apart) while strictly enforcing that the distances between neighboring points remain unchanged. This "unfolding" process, implemented via semidefinite programming, discovers the true degrees of freedom in complex datasets by treating the local neighborhood connections like a rigid but jointed framework that can be flattened but not stretched.
Bridging Linear and Non-linear Projections
While non-linear methods like MVU are powerful, they can be computationally expensive. He et al. introduced Locality Preserving Projections (LPP) to provide a linear approximation to these non-linear effects.
LPP works by constructing an adjacency graph that represents the local neighborhood structure. It then finds a linear projection that minimizes the weighted distance between neighboring points in the reduced space. This approach prioritizes the preservation of the manifold's topological structure over global variance, preventing the overlapping of distinct clusters that often occurs in traditional linear projections. As noted by Gao et al., this is particularly effective for multimedia data, which naturally resides on lower-dimensional manifolds within high-dimensional ambient spaces.
Supervised Locality and Discriminative Power
Locality preservation is not limited to unsupervised learning. When labels are available, the definition of a "neighbor" can be refined to improve classification. Cai et al. proposed Locality Sensitive Discriminant Analysis (LSDA), which modifies the objective: nearby points with the same label are pulled together, while nearby points with different labels are pushed apart.
Similarly, Nie et al. developed Neighborhood MinMax Projections (NMMP), arguing that focusing on local neighborhood structures rather than global class averages allows the model to resolve complex, non-Gaussian distributions. By respecting the local geometry within each class, these methods prevent the manifold from "breaking" during reduction, leading to cleaner class separation than global methods like Linear Discriminant Analysis (LDA).
Robustness and Integration
Modern research has extended these concepts to handle noise and integrate dimensionality reduction with other tasks. Zhang et al. introduced Robust Manifold Matrix Factorization (RMMF), which uses manifold regularization to ensure that the intrinsic non-linear geometry is preserved even when the data contains outliers. By performing feature extraction and clustering simultaneously, the authors show that maintaining local geometric relationships is critical for accurately clustering datasets with non-linear distributions.
This focus on local structure has proven essential across diverse domains:
- LiDAR Processing: Huang et al. use locality-preserving reduction to ensure that points with similar local geometric contexts in 3D urban environments remain clustered in feature space.
- Causal Inference: Zhang et al. apply these projections to ensure that samples that are close in high-dimensional feature space remain neighbors for accurate matching.
- Feature Learning: Gu et al. demonstrate that preserving local geometric structures captures finer semantic nuances in document clustering than methods prioritizing global variance.
Ultimately, manifold-aware dimensionality reduction works because it acknowledges a fundamental truth of high-dimensional data: the global map may be curved and complex, but the local neighborhood provides a reliable, linear anchor for understanding the data's true structure.
Key Foundational Work
Unsupervised Learning of Image Manifolds by Semidefinite Programming (2006) — contributed 2 concept elements to this direction.
Locality Preserving Projections (2003) — contributed 1 concept elements to this direction.
An Introduction to Nonlinear Dimensionality Reduction by Maximum Variance Unfolding (2006) — contributed 1 concept elements to this direction.