In machine learning, we often assume that high-dimensional data—whether it’s a stream of video pixels, a complex protein interaction network, or a chaotic fluid simulation—is actually governed by a much smaller set of hidden variables. We call this the "latent space." However, a persistent "Goldilocks" problem plagues the field: how do we find a latent space that is just right?
If the latent space is too small, the model suffers from information loss, failing to capture the "physics" of the system. If it is too large, the model becomes over-parameterized, capturing redundant noise and losing interpretability. Historically, researchers like Niklas Wahlström and Thomas B. Schön noted that specifying this dimensionality manually requires deep prior knowledge of the system's complexity, which we rarely have for unknown physical processes Learning deep dynamical models from image pixels.
Recent research has shifted away from manual tuning toward mechanisms that allow models to autonomously discover their own intrinsic dimensionality.
The Geometry of the Bottleneck
The search for intrinsic dimensionality is essentially a search for the "manifold"—the low-dimensional surface upon which the data actually lives. Early attempts at this involved "greedy" pruning. David DeMers and Garrison W. Cottrell showed that by penalizing the variance of hidden units in an autoencoder, a model could effectively "shut off" unnecessary dimensions Non-Linear Dimensionality Reduction.
More modern approaches use sophisticated priors to achieve this "soft" pruning. For example, Andreas Geiger and Raquel Urtasun introduced rank priors for Gaussian Process Latent Variable Models (GPLVM). By penalizing the singular values of the latent space during optimization, the model automatically collapses redundant dimensions, avoiding the local minima that trap traditional methods like PCA Rank priors for continuous non-linear dimensionality reduction.
Automatic Relevance Determination
A recurring mechanism for solving the dimensionality problem is Automatic Relevance Determination (ARD). ARD acts as a mathematical filter that determines which latent dimensions are "relevant" to the output and which are just noise.
In the context of deep generative models, Gunnar Ratsch and Theofanis Karaletsos used ARD to move away from computationally expensive manual searches. Instead of training dozens of models with different bottleneck sizes, ARD allows a single model to learn which latent variables are necessary to represent the data Automatic Relevance Determination For Deep Generative Models. This concept has been extended to non-Gaussian data by Dacheng Tao, who integrated ARD into exponential family PCA to automatically determine the number of principal components Simple Exponential Family PCA.
Growing and Balancing Models
Rather than starting with a large space and pruning it, some researchers propose models that "grow" to fit the data. Hugo Larochelle and Marc-Alexandre Côté developed the Infinite Restricted Boltzmann Machine (iRBM). By introducing an ordering to hidden units and an incremental energy penalty, the iRBM adds capacity only when the data demands it, effectively removing the hidden-unit hyperparameter entirely An Infinite Restricted Boltzmann Machine.
In physics-informed modeling, the stakes are higher: the latent space must not only be the right size but also respect dynamical stability. Julius Aka introduced Balanced Neural ODEs (B-NODEs), which use an information bottleneck to "balance" reconstruction accuracy against model complexity. This allows the model to compress hundreds of physical states into a handful of latent variables that capture the "slow" and "fast" processes of a system Balanced Neural ODEs: nonlinear model order reduction and Koopman operator approximations.
Decoding the Geometry of Data
Perhaps the most surprising recent insight is that generative models might already "know" the intrinsic dimension of the data they study. Jan Pawel Stanczuk and Georgios Batzolis proved that the score function of a trained diffusion model actually encodes the geometry of the data manifold. By analyzing this score function, one can estimate the intrinsic dimension of a dataset more accurately than with traditional statistical benchmarks Diffusion Models Encode the Intrinsic Dimension of Data Manifolds.
Similarly, in the study of chaotic systems, Jason J. Brunton and colleagues have used autoencoders to find "topological conjugacies"—coordinate systems where messy, high-dimensional chaos becomes simple, linear math. The challenge remains aligning the latent dimensionality with the system's inherent fractal dimension to avoid redundant information Deep learning of conjugate mappings.
Why It Works: The Principle of Parsimony
The underlying intuition across all these papers is the principle of parsimony: the simplest explanation that fits the data is usually the most robust. Whether through jerk regularization to ensure smooth latent trajectories Smooth and Sparse Latent Dynamics in Operator Learning with Jerk Regularization or non-parametric priors that infer cluster counts Bayesian Matrix Factorization with Side Information and Dirichlet Process Mixtures, the field is moving toward models that treat dimensionality not as a fixed setting, but as a discoverable property of the universe.