Most classic dimensionality reduction (DR) algorithms are transductive: they find a low-dimensional home for a specific set of points, but they don't learn a general rule for how to get there. If you have a 2D map of 10,000 points created by t-SNE or Isomap and a 10,001st point arrives, you are stuck. You cannot simply "drop" the new point onto the map; you must re-run the entire, often computationally expensive, optimization from scratch.
This is known as the out-of-sample problem. A growing body of research focuses on transforming these "static" embeddings into parametric models—functions that can project new, unseen data points instantly.
The Transductive Trap
In non-parametric DR, the "model" is just a list of coordinates. Methods like Isomap, Laplacian Eigenmaps, and traditional t-SNE optimize the positions of points in a low-dimensional space to preserve some property of the high-dimensional space (like geodesic distance or local neighborhood probability).
As Christopher K. I. Williams noted in On a Connection between Kernel PCA and Metric Multidimensional Scaling, the lack of a direct functional mapping is a significant drawback. Without a formula, these methods cannot be used in real-time systems, streaming data environments, or large-scale retrieval tasks where data arrives incrementally.
From Coordinates to Functions
The solution is to replace the direct optimization of coordinates with the optimization of a mapping function $f(x; \theta)$. Instead of moving points, we tune the parameters $\theta$ of a model—usually a neural network—so that the network learns to produce the desired embedding.
Early attempts at this include NeuroScale, developed by David Lowe and Michael E. Tipping in NeuroScale: Novel Topographic Feature Extraction using RBF Networks, which used Radial Basis Function (RBF) networks to mimic Multidimensional Scaling. Later, Laurens van der Maaten introduced parametric t-SNE in Learning a Parametric Embedding by Preserving Local Structure, showing that a deep neural network could be trained to minimize the t-SNE loss function directly.
This shift from transductive to inductive learning provides three major benefits:
- Instant Inference: New data points can be projected using a simple forward pass through the network.
- Scalability: You can train on a representative subset (or "exemplars") and project the rest, as explored by Martin Renqiang Min in Parametric t-Distributed Stochastic Exemplar-centered Embedding.
- Compression: High-dimensional features (like face representations) can be squeezed into their "intrinsic dimensionality" while remaining functional for downstream tasks, a concept demonstrated by Sixue Gong in On the Intrinsic Dimensionality of Face Representation.
Bridging the Gap with Kernels and Eigenfunctions
Before deep learning became the standard "function approximator," researchers looked for mathematical bridges to extend spectral methods. In Out-of-Sample Extensions for LLE, Isomap, MDS, Eigenmaps, and Spectral Clustering, Yoshua Bengio and colleagues provided a unified framework by treating these algorithms as learning the eigenfunctions of a kernel.
By framing DR as a kernel problem, one can use the Nyström method to interpolate the position of a new point based on its similarity to the training points. This "kernelization" strategy is still vital today; for instance, Jan Quan and Johan Suykens recently revisited PCA through duality in Rethinking PCA Through Duality to create robust, kernelizable variants that handle new data in noisy environments.
Overcoming the Parametric Penalty
While parametric models solve the out-of-sample problem, they often struggle to capture the "crisp" local structures that non-parametric methods excel at. Neural networks tend to smooth out the embedding space, sometimes blurring distinct clusters into a single mass.
Recent work has focused on "restoring" this lost detail. Haiyang Huang and Cynthia Rudin introduced ParamRepulsor in Navigating the Effect of Parametrization for Dimensionality Reduction, which uses hard negative mining to force the network to maintain clear boundaries between clusters. Similarly, Chien-Hsun Lai and Yu-Shuen Wang identified that "exploding gradients" often ruin parametric training, proposing gradient clipping in Facilitate the Parametric Dimension Reduction by Gradient Clipping to stabilize the learning of t-SNE and UMAP mappings.
Modern Frontiers: Explainability and Geometry
The move toward parametric models has opened the door to Explainable AI (XAI). Because a parametric model like EVNet, developed by Zelin Zang in EVNet: An Explainable Deep Network for Dimension Reduction, uses an explicit architecture (like a Lasso network), researchers can inspect the weights to see which high-dimensional features are responsible for a specific cluster in the 2D visualization.
Furthermore, frameworks like Deep Manifold Transformation (DMT) by Stan Z. Li in Deep Manifold Transformation for Nonlinear Dimensionality Reduction use "Local Geometry-Preserving" constraints across neural network layers. This ensures that the manifold structure is not just preserved at the output, but is respected throughout the entire transformation process, leading to more robust and consistent representations for complex data like single-cell sequences or high-rate time series.