When compressing high-dimensional data—whether it represents fluid dynamics, climate simulations, or neural network activations—our default tool is often a linear projection like Principal Component Analysis (PCA) or Proper Orthogonal Decomposition (POD). These methods assume that the data lies on or near a flat hyperplane.
But what happens when the underlying phenomena are inherently dynamic and moving, such as a sharp flame front or a traveling wave? As a wave propagates across a spatial domain, its trajectory in high-dimensional state space traces out a highly curved path. To approximate this curved path using flat planes, linear methods require an excessive number of basis vectors. This limitation is known as the Kolmogorov barrier. As noted by Rudy Geelen et al. in Learning physics-based reduced-order models from data using nonlinear manifolds, advection-dominated or transport-dominated problems cause traditional linear methods to smooth over high-frequency features or fail entirely unless given a prohibitively large latent space.
The Core Mechanism: Quadratic Enrichment
To overcome this without resorting to the opaque complexity and heavy computational overhead of deep neural networks, researchers have turned to quadratic manifolds. The core intuition is simple: instead of forcing the latent space to map linearly to the high-dimensional space, we introduce a second-order polynomial correction.
In a standard linear reduction, a high-dimensional state $x$ is reconstructed from a low-dimensional latent vector $r$ using a linear basis matrix $U$:
$$x \approx Ur$$
A quadratic manifold expands this formulation by adding a nonlinear correction term composed of quadratic combinations of the reduced coordinates. As formulated by Paul Schwerdtner et al. in Online learning of quadratic manifolds from streaming data for nonlinear dimensionality reduction and nonlinear model reduction, the mapping becomes:
$$x \approx Ur + V(r \otimes r)$$
where $V$ represents a matrix that maps the Kronecker product of the reduced coordinates back into the high-dimensional space.
By shifting the approximation power from first-order linear to second-order quadratic, the model can capture complex curvatures using significantly fewer latent dimensions. For instance, Rudy Geelen et al. demonstrated in Learning latent representations in high-dimensional state spaces using polynomial manifold constructions that incorporating these low-order polynomial terms can reduce reconstruction errors by up to two orders of magnitude compared to purely linear decompositions when modeling nonlinear wave dynamics.
Why It Works: Geometric Intuition
To understand why this quadratic correction is so powerful, consider the geometry of a simple circle or a curved trajectory. A linear model can only approximate a curve through a sequence of segmented flat patches, requiring many dimensions to avoid fragmentation. A quadratic function, however, natively possesses the mathematical vocabulary to express curvature.
This geometric efficiency manifests across several distinct domains:
- Tracking Moving Waves: In physical systems featuring solitons or moving fronts, the quadratic terms allow the reduced coordinates to smoothly parameterize the translation of features across space. Instead of needing a new linear basis vector for every incremental shift in a wave's position, the quadratic manifold bends to follow the wave's trajectory (Rudy Geelen et al. in Operator inference for non-intrusive model reduction with quadratic manifolds).
- Uncovering Concept Clusters: In the field of mechanistic interpretability, Thomas Dooms et al. developed Bilinear Autoencoders in Finding Manifolds With Bilinear Autoencoders. They discovered that neural networks often compose information into circular or triangular concept manifolds. While linear models fragment these structures, quadratic primitives preserve their global geometry cleanly.
- Robust Clustering and Outlier Detection: Rather than defining subspaces as flat hyperplanes, data can be modeled as the intersection of multiple quadric hypersurfaces (the zero-sets of quadratic polynomials). As shown by Fedor Pavutnitskiy et al. in Quadric Hypersurface Intersection for Manifold Learning in Feature Space, this approach captures intricate clusters and curvatures while remaining highly scalable.
Algorithmic Frameworks and Scaling
Implementing quadratic manifolds requires balancing expressivity with computational efficiency, as the number of quadratic combinations grows quadratically with the latent dimension. Researchers have developed several strategies to make this tractable:
- Quasi-Quadratic Selection: To manage the complexity of the $r \otimes r$ term, Rudy Geelen et al. proposed selecting only a subset of important quadratic terms, creating a "quasi-quadratic manifold" that optimizes the tradeoff between accuracy and computational cost.
- Quadratic Matrix Factorization (QMF): Extending traditional matrix factorization, Hengchao Chen et al. introduced Quadratic Matrix Factorization With Applications to Manifold Learning, utilizing an alternating minimization algorithm to faithfully reconstruct manifold contours and improve data denoising.
- Online and Greedy Learning: For petabyte-scale simulation data, batch processing is impossible. In Online learning of quadratic manifolds from streaming data for nonlinear dimensionality reduction and nonlinear model reduction, the authors introduced an online greedy method that incrementally updates a truncated SVD to build the quadratic manifold in a single pass.
- Sparse Sensing: In practical scenarios, we often need to reconstruct a high-dimensional field from just a few physical sensors. Paul Schwerdtner et al. developed Quadratic Manifold Sparse Regression in Empirical sparse regression on quadratic manifolds, demonstrating that quadratic manifolds allow full field reconstruction from minimal sensor counts where linear methods fail.
- Galerkin Schemes: To ensure stability when solving partial differential equations on these manifolds, Paul Schwerdtner et al. integrated quadratic decoders with Neural Galerkin schemes in Nonlinear model reduction with Neural Galerkin schemes on quadratic manifolds, ensuring locally unique and stable solutions over time.
- Hierarchical Coding: Even in sparse representation frameworks, Wei Xiao et al. demonstrated in Two-Layers Local Coordinate Coding that a two-layer anchor point architecture elevates local coordinate coding to second-order quadratic approximation power, drastically reducing error bounds for noisy or scarce data.
Key Foundational Work
Operator inference for non-intrusive model reduction with quadratic manifolds (2022) — contributed 4 concept elements to this direction.
Quadric Hypersurface Intersection for Manifold Learning in Feature Space (2022) — contributed 2 concept elements to this direction.
Two-Layers Local Coordinate Coding (2015) — contributed 1 concept elements to this direction.