A Deep Generative Approach to Stratified Learning

Randy Martinez, Rong Tang, Lizhen LinarXiv 2026 · arXiv

TL;DR

This paper moves beyond the traditional manifold hypothesis to tackle "stratified spaces"—complex datasets made up of multiple intersecting manifolds of different dimensions. The authors introduce two novel deep generative frameworks (a dimension-aware mixture of VAEs and a score-based diffusion model) that can accurately learn data distributions on these spaces while consistently estimating both the number of strata and their intrinsic dimensions.

What Problem Does This Paper Address?

Most modern machine learning algorithms rely on the manifold hypothesis, which assumes high-dimensional data sits on or near a single, low-dimensional smooth manifold. However, real-world data—like natural images or molecular dynamics configurations—is often much messier. It frequently forms stratified spaces, which are unions of multiple manifolds (strata) that possess different dimensions and can intersect.

Learning on stratified spaces is notoriously difficult. The intersections create geometric singularities where standard manifold properties (like smoothness and positive reach) break down. Classical dimension estimation and representation learning tools struggle or fail entirely near these intersections, especially when ambient noise is introduced. This paper addresses the lack of robust generative models and statistical guarantees for learning distributions on these complex, singular geometric structures.

Methods

The authors propose two distinct deep generative frameworks to handle stratified learning:

  1. Sieve MLE (Likelihood-Based Approach): This method uses a dimension-aware Mixture-of-Experts (MoE) architecture implemented via Variational Autoencoders (VAEs). To handle the singularities at intersections, the authors use a mathematical technique called excision—partitioning the space into regular regions with positive reach and isolating the intersection zones to control approximation errors.
  2. Forward-Backward Diffusion Framework: Instead of explicitly partitioning the space, this approach leverages the power of diffusion models. The forward process incrementally injects Gaussian noise, which naturally smooths out the sharp singularities of the stratified space. The backward process uses a piecewise ReLU neural network to approximate the score field.
  3. Geometric Dimension Estimator (Algorithm 1): By analyzing the local geometry of the learned score field at small diffusion time scales ($t \to 0^+$), the authors show that the score vector aligns normally to the strata. They develop an algorithm using Singular Value Decomposition (SVD) on sampled score vectors to estimate the local intrinsic dimension (LID) and cluster the data into its constituent strata.

Datasets

The authors evaluated their approaches using both synthetic geometric constructs and real-world scientific datasets:

  • Synthetic Manifolds:
  • An intersecting circle (1D) and sphere (2D) embedded in $\mathbb{R}^{50}$ under varying noise levels ($\sigma_* = 0, 0.02, 0.05$).
  • A disjoint union of four manifolds (1D helix, 2D torus, 4D sphere, and 7D sphere) translated in $\mathbb{R}^{50}$.
  • A union of a circle and a 2D plane.
  • A union of a 1D helix and a 2D Swiss roll.
  • Real-World Molecular Dynamics:
  • Butane: A small molecule represented as a 12-dimensional vector.
  • Alanine Dipeptide: A standard benchmark dataset for local intrinsic dimension estimators, focusing on heavy atoms to avoid high-frequency hydrogen noise.

Key Insights and Ideas

  • Noise as a Blessing, Not Just a Curse: In the likelihood framework, ambient noise is a theoretical necessity to prevent singular distributions from collapsing. While too much noise obscures geometry, a moderate amount of noise (or artificial data perturbation) stabilizes the maximum likelihood estimation. In diffusion models, the built-in Gaussian smoothing acts as an automatic regularizer, making it exceptionally well-suited for highly singular spaces.
  • Dominance of Low Dimensions at Intersections: The paper reveals a fascinating geometric property of the score field: near singularities where multiple strata intersect, the limiting behavior of the score function is dominated by the normal space of the lowest-dimensional participating stratum.
  • Complementary Strengths: The two frameworks serve different regimes. The Sieve-MLE approach excels when noise is moderate and a direct deconvolution (recovering the noiseless distribution) is desired. Diffusion models shine in near-singular or completely noiseless settings because they don't require artificial noise injection to remain stable.

Key Results

  • Robustness to Noise: On the synthetic circle-and-sphere dataset with ambient noise ($\sigma_* = 0.05$), classical baselines like Local PCA and the Levina-Bickel MLE failed completely, dropping to 0% accuracy in identifying the correct dimensions. In contrast, the proposed diffusion-based estimator (Algorithm 1) maintained high accuracy (~90%) across various time windows.
  • Multi-Manifold Estimation: On the 4-manifold mixture embedded in $\mathbb{R}^{50}$ with noise, Algorithm 1 achieved an overall accuracy of 85.74%, vastly outperforming Local PCA (25.79%) and Levina-Bickel MLE (19.57%).
  • Real-World Validation: For the Butane dataset, classical estimators erroneously predicted an intrinsic dimension between 6 and 8. The proposed diffusion estimator correctly identified a majority of 1D and 2D structures, matching physical domain knowledge. For Alanine Dipeptide, the algorithm correctly assigned over half the points to a 2D manifold, validating the widely accepted hypothesis that its underlying structure is a 2D torus.

Limitations and Future Work

  • Simplified Noise Models: The current framework assumes a uniform, isotropic noise level across all strata. Real-world data might exhibit varying or anisotropic noise depending on the specific stratum.
  • Strata Scaling: The proofs assume a fixed number of strata ($K$) and non-negligible mixing weights ($\omega_{min}$). The authors note that handling non-parametric mixtures where $K \to \infty$ or where some strata have arbitrarily small weights will require more advanced detection analysis.
  • Computational Complexity: Training diffusion models and performing SVD on large score matrices can be computationally heavy, leaving room for future optimization of the algorithmic pipeline.

Summary updated

Abstract

While the manifold hypothesis is widely adopted in modern machine learning, complex data is often better modeled as stratified spaces -- unions of manifolds (strata) of varying dimensions. Stratified learning is challenging due to varying dimensionality, intersection singularities, and lack of efficient models in learning the underlying distributions. We provide a deep generative approach to stratified learning by developing two generative frameworks for learning distributions on stratified spaces. The first is a sieve maximum likelihood approach realized via a dimension-aware mixture of variational autoencoders. The second is a diffusion-based framework that explores the score field structure of a mixture. We establish the convergence rates for learning both the ambient and intrinsic distributions, which are shown to be dependent on the intrinsic dimensions and smoothness of the underlying strata. Utilizing the geometry of the score field, we also establish consistency for estimating the intrinsic dimension of each stratum and propose an algorithm that consistently estimates both the number of strata and their dimensions. Theoretical results for both frameworks provide fundamental insights into the interplay of the underlying geometry, the ambient noise level, and deep generative models. Extensive simulations and real dataset applications, such as molecular dynamics, demonstrate the effectiveness of our methods.
BibTeX
@misc{martinez2026deep,
  author = {Randy Martinez and Rong Tang and Lizhen Lin},
  title = {{A Deep Generative Approach to Stratified Learning}},
  year = {2026},
  eprint = {2604.10650},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2604.10650},
}

Loading related content...

Figure