The foundational assumption of modern representation learning is the manifold hypothesis: the idea that high-dimensional real-world data (such as images, text embeddings, or physical measurements) actually concentrates around a much lower-dimensional, continuous surface.
For decades, machine learning algorithms have relied on a crucial mathematical luxury within this hypothesis: the assumption of smoothness. If a data manifold is perfectly smooth and differentiable, it can be treated locally as a flat Euclidean space. This geometric regularity provides powerful guarantees. As shown by Hariharan Narayanan et al. in Sample Complexity of Testing the Manifold Hypothesis, fitting a low-dimensional manifold to data scales with its intrinsic properties—like volume and curvature—allowing models to bypass the curse of dimensionality.
However, real-world data rarely behaves like a pristine, smooth rubber sheet. Instead, empirical data distributions are frequently riddled with singularities: regions where the manifold pinches, self-intersects, changes dimensionality abruptly, or forms sharp, non-differentiable edges. A growing body of research focuses on understanding what happens when data violates the smooth manifold assumption, why these violations break standard models, and how we can design algorithms to handle geometric irregularities.
The Reality of Real-World Data: Where Smoothness Breaks
In practice, assuming that data lies on a globally smooth manifold is often an oversimplification. Complex data spaces naturally generate structural anomalies:
- Token Embeddings: While researchers frequently use visualization tools like t-SNE or UMAP under the assumption of underlying smoothness, Michael Robinson et al. demonstrate in Token Embeddings Violate the Manifold Hypothesis that large language model token spaces are filled with mathematical jagged edges, such as cusps and local dimension changes.
- Physical Systems: In material science and data-driven mechanics, abrupt physical transitions (like material fractures or phase changes) create discontinuous jumps. The authors of Manifold embedding data-driven mechanics note that these phenomena directly violate the smooth Riemannian surface assumptions embedded in traditional modeling frameworks.
When data contains these self-intersections or sharp boundaries, the mathematical guarantees underpinning many common machine learning workflows begin to collapse.
Why Smoothness Violations Break Machine Learning Models
To understand why singularities cause models to fail, we must look at the mechanics of how algorithms interact with local geometry. Smooth manifold learning relies on properties like positive reach (a bound on how sharply a surface can curve before intersecting itself) and local linearity. When a singularity occurs, these properties disappear, causing failures across three major regimes:
1. Distorted Geometric Reconstructions
Spectral methods and dimensionality reduction techniques often map data by simulating random walks across a neighborhood graph. As explored by Boaz Nadler et al. in Diffusion maps, spectral clustering and reaction coordinates of dynamical systems, when data contains spikes or non-differentiable regions, the underlying differential operators fail to converge properly. The random walk gets "trapped" or distorted by the singularity, leading to inaccurate low-dimensional representations.
2. Optimization Failures
In non-convex optimization frameworks—such as matrix factorization for large-scale semidefinite programs—smoothness is required to guarantee that local search methods find global optima. Diego Cifuentes et al. point out in Polynomial time guarantees for the Burer-Monteiro method that the presence of singularities or non-smooth boundaries introduces structural failure modes that violate standard constraint qualifications, stripping away theoretical convergence speed guarantees.
3. Generative Model Instability
In diffusion models, the sudden emergence of data features during the reverse generation process behaves like a geometric phase transition. This transition occurs when generating particles enter a "tubular neighborhood" where they can uniquely project onto the target data manifold. As shown by Manato Yaguchi et al. in The Geometry of Phase Transitions in Diffusion Models: Tubular Neighbourhoods and Singularities, if the target manifold contains sharp edges or corners, these tubular neighborhoods become mathematically ill-defined, explaining some of the unpredictable instabilities observed during generation.
Diagnosing and Measuring Geometric Irregularity
If real-world data is inherently non-smooth, how can we detect where the manifold hypothesis fails? Standard Topological Data Analysis (TDA) tools, like persistent homology, often "smooth over" these critical structural details because they treat data through a uniform lens.
To capture these hidden anomalies, researchers have introduced multi-scale geometric diagnostics:
- Euclidicity: In TOAST: Topological Algorithm for Singularity Tracking, Julius Von Rohrscheidt et al. introduce a metric called Euclidicity. By measuring how much a local neighborhood deviates from a perfectly flat Euclidean patch across multiple scales, they provide a direct way to track and isolate structural singularities.
Smooth Manifold (High Euclidicity) Singularity / Pinch Point (Low Euclidicity)
─────────────────── ─────── ───────
│ . . . . │ \ . /
│ .
* . . │ \*/ <- Local flatness
─────────────────── / \ breaks down
Locally looks like flat space Data intersects or changes dim
- Intersection Homology: Rather than treating a dataset as a single uniform space, Bastian Rieck et al. leverage stratified spaces in Persistent Intersection Homology for the Analysis of Discrete Data. By applying a "perversity function" that filters out or highlights singular intersections, the authors show that a significant quantitative divergence between standard persistent homology and intersection-based homology serves as a clear diagnostic indicator that the smooth manifold hypothesis has been violated.
Designing for Non-Smooth Geometries
Recognizing the limitations of the smooth manifold assumption allows for the design of architectures that explicitly accommodate geometric complexity. Instead of forcing rough data into smooth mathematical boxes, modern approaches adapt to the terrain:
Fractal Partitioning
Real-world data often exhibits messy, non-integer fractal dimensions rather than clean, differentiable structures. To address this, Ingo Steinwart et al. propose a data-dependent partitioning scheme in Intrinsic Dimension Adaptive Partitioning for Kernel Methods. By dividing the input space based on local fractal dimensions rather than assuming a smooth global manifold, they achieve optimal learning rates for kernel methods on highly irregular datasets.
Expansive Injective Flows
When modeling probability distributions on manifolds using normalizing flows, sharp corners and self-intersections typically cause numerical blowups. Michael Anthony Puthawala addresses this in Universal Joint Approximation of Manifolds and Densities by Simple Injective Flows, proving that expansive injective flows can universally approximate both the geometry and density of non-smooth manifolds, provided the embedding dimension is sufficiently large to untangle topological knots and self-intersections.
By moving beyond the comforting illusion of global smoothness, these frameworks allow machine learning models to maintain their theoretical guarantees and structural robustness, even when confronted with the jagged reality of empirical data.
Key Foundational Work
Fiber homology and orientability of maps (1980) — contributed 1 concept elements to this direction.
On an existence and uniqueness theory for nonlinear differential-algebraic equations (1991) — contributed 1 concept elements to this direction.