Classical manifold learning and spectral algorithms are "transductive" by nature. If you run Spectral Clustering or t-SNE on a dataset, the algorithm produces a set of coordinates for those specific points. However, if a single new data point arrives, these methods typically break: they have no inherent "rule" for where to place the new point without re-running the entire, often cubic-complexity, optimization.
A growing body of research solves this "out-of-sample" problem by replacing discrete point-embeddings with parametric functional mappings. By training a neural network to approximate the eigenfunctions of a kernel or the eigenvectors of a graph Laplacian, researchers are transforming static embeddings into dynamic, scalable models capable of real-time inference.
The Out-of-Sample Problem
In traditional manifold learning, the embedding is the result of an optimization over a fixed set of variables—one for each data point. As Laurens van der Maaten notes in the development of Parametric t-SNE, this makes the models non-parametric and unable to generalize. To add a new point, one would traditionally need to use expensive interpolation like Nyström extensions or re-optimize the entire system.
The shift toward inductive models changes the objective. Instead of optimizing the coordinates $Y$, we optimize the weights $\theta$ of a neural network $f_\theta(x) = y$. Once trained, this network acts as a reusable mathematical function. David Lowe and Michael E. Tipping demonstrated this early on with NeuroScale, using Radial Basis Function (RBF) networks to preserve inter-point distances. This allows the model to place novel clusters in their correct relative positions within a lower-dimensional space simply by passing them through the network.
Bypassing the Cubic Bottleneck
Spectral methods are mathematically elegant but computationally "divas." Calculating the eigenvectors of a graph Laplacian or a kernel matrix requires $O(n^3)$ complexity, which is impossible for datasets with millions of nodes.
Researchers like Uri Shaham and Ronen Basri address this in SpectralNet by reformulating spectral clustering as a functional approximation task. Instead of an explicit eigendecomposition, they use a neural network with a specialized orthogonalization layer. This allows the model to learn the "spectral shape" of the data using stochastic mini-batch optimization, which scales linearly with the number of points.
Similarly, Zhijie Deng and Jun Zhu introduced NeuralEF, which deconstructs kernels by learning their principal eigenfunctions. By treating eigendecomposition as a series of asymmetric optimization problems, they enable kernel methods to scale to massive datasets that were previously unreachable.
Neural Networks as Eigenfunction Approximators
The core intuition behind this research direction is that neural networks are universal approximators for the eigenfunctions of operators.
- Graph Dynamics: In G-SPARC, Yahel Jacobs and colleagues show that a network can learn to map node features into a spectral embedding space. This allows a Graph Neural Network to predict the structural position of a "cold-start" node—one that has no existing edges—by inferring where it should sit on the graph's manifold.
- Diffusion Manifolds: Gal Mishne and Alexander Cloninger developed Diffusion Nets, which train autoencoders to learn Diffusion Maps. By enforcing an eigenvector constraint on the latent space, the network learns a functional mapping that solves the "pre-image problem"—the difficult task of mapping a point from the low-dimensional embedding back into the original high-dimensional space.
- Clustering as Inference: Methods like Neural Normalized Cut by Rong Xiao and Chun-Guang Li transform clustering from a global optimization problem into a simple forward pass. This makes spectral clustering nearly instantaneous at test time, even for millions of samples.
Bridging the Local-Global Gap
A persistent challenge in parametric dimensionality reduction is that neural networks often struggle to capture the fine-grained local structures that non-parametric methods (like UMAP) excel at. Jungeum Kim and Xiao Wang address this in iGLoMAP by using a two-step "particle-based" training. They first update embedding coordinates transductively and then train a deep network to approximate those positions. This hybrid approach ensures the network learns to preserve both the "forest" (global geometry) and the "trees" (local clusters).
By parameterizing these classical geometric ideas, the field is moving away from viewing data as a static collection of points and toward viewing it as a continuous manifold that can be navigated by a learned function. This transition not only solves the scalability issues of the past but also allows spectral features to be integrated directly into modern deep learning pipelines.