When dealing with high-dimensional data, machine learning models frequently run into the curse of dimensionality: as the number of features grows, the data becomes sparse, distances distort, and computational costs explode. To bypass this, a foundational strategy across the field is to project the data into a lower-dimensional linear subspace or manifold where the essential structure of the data is preserved.
However, this strategy introduces a critical, often unresolved question: How many dimensions should the target subspace have?
Choosing the dimensionality ($d$) of a projected subspace is rarely a matter of simple aesthetics. It represents a fundamental trade-off between information preservation and noise reduction. Historically, practitioners have relied on manual tuning or rigid heuristics to select this value. A growing body of research focuses on understanding the sensitivity of models to this parameter and developing automated, data-driven methods to select the optimal subspace dimensionality.
The Geometry of the Subspace Dilemma
To understand why selecting the right dimensionality is difficult, consider the geometry of a projection. If we project high-dimensional data onto a subspace that is too small, we suffer from severe information loss. In the context of representation learning, this can manifest as dimensional collapse, where the model fails to utilize the full capacity of the embedding space. Li Jing et al. explore this phenomenon in Understanding Dimensional Collapse in Contrastive Self-supervised Learning, demonstrating that when the subspace size is poorly calibrated, representations shrink to occupy only a tiny fraction of the available space, discarding vital discriminative features.
Conversely, making the subspace too large introduces its own set of failures. Instead of capturing clean, low-rank structures, an over-dimensioned subspace retains excessive noise and irrelevant variances. For instance, in tensor decomposition and denoising, Qinqing Zheng et al. show in Interpolating Convex and Non-Convex Tensor Decompositions via the Subspace Norm that retaining too many singular vectors directly degrades the accuracy of the recovered data by letting noise bleed into the refined estimation.
High-Dimensional Data
│
├─► Too Few Dimensions ──► Dimensional Collapse & Information Loss
│
└─► Too Many Dimensions ─► Noise Retention & Computational Explosion
A Universal Bottleneck Across Domains
The challenge of subspace selection is not isolated to a single domain; it is a recurring bottleneck across diverse machine learning paradigms:
- Distance Metric Learning & Domain Adaptation: When transferring knowledge between completely different feature spaces (e.g., text tags to images), models project data into a shared latent subspace. Yong Luo et al. note in General Heterogeneous Transfer Distance Metric Learning via Knowledge Fragments Transfer that the rank of this mapped subspace must typically be specified manually, creating a major optimization hurdle. Similarly, Raghuraman Gopalan relies on empirical heuristics like signal energy preservation thresholds in Bridging Heterogeneous Domains With Parallel Transport For Vision and Multimedia Applications to determine generative subspace sizes, which lacks theoretical guarantees across varying data distributions.
- Statistical Inference & Hypothesis Testing: In high-dimensional two-sample testing, random projections are used to make classical tests computationally tractable. However, as shown by Miles Lopes et al. in A More Powerful Two-Sample Test in High Dimensions using Random Projection and Jie Wang et al. in Two-sample Test using Projected Wasserstein Distance, the optimal projection dimension depends heavily on the unknown relationship between sample size and data dimensionality. Without data-driven selection, these tests risk a severe drop in statistical power.
- Modern Deep Learning & Optimization: Even in cutting-edge applications like black-box visual prompting (Zhaogeng Liu et al. in Query Efficient Black-Box Visual Prompting with Subspace Learning) or second-order optimization (Hadar Sivan et al. in FOSI: Hybrid First and Second Order Optimization), tracking the right number of eigenvectors or subspace dimensions remains a manual tuning burden that directly impacts convergence speed and query efficiency.
Moving Toward Automated and Adaptive Solutions
To move away from computationally expensive trial-and-error methods like $k$-fold cross-validation (a limitation discussed by Johan Suykens et al. in Kernel PLS variants for regression), researchers are designing algorithms that can intrinsically determine or adaptively navigate subspace dimensions.
1. Theoretical Bounds and Approximation Targets
One approach is to establish formal mathematical frameworks that pre-calculate the required dimensionality based on a target error tolerance. For instance, in Common component analysis for multiple covariance matrices, Huahua Wang et al. derive theoretical lower and upper bounds for their objective function. This allows practitioners to systematically choose the optimal dimensionality needed to meet a specific target approximation error, replacing pure guesswork with geometric guarantees.
2. Parameter-Free Formulations
Another direction eliminates the dimensionality hyperparameter entirely by forcing the algorithm to infer the data's structural rank from the underlying tensor or matrix properties. Rudolf Fleischer demonstrates this in Multilinear Tensor-Based Non-parametric Dimension Reduction for Gait Recognition, introducing a parameter-free framework that automatically determines the optimal dimensionality for data reduction while preserving multi-dimensional structural dependencies.
3. Dynamic Manifold Exploration
Rather than treating the subspace dimension as a static number chosen before training, some frameworks treat the selection process as a dynamic exploration problem on geometric manifolds. In Grassmann Stein Variational Gradient Descent, Xing Liu utilizes a stochastic diffusion process on a Grassmann manifold (the space of all linear subspaces). By adaptively navigating this manifold, the algorithm dynamically projects particles onto the most informative subspaces, allowing the model to adjust its focus as the complexity of the target distribution changes.
Conclusion
Automated subspace dimensionality selection represents a shift from heuristic engineering to principled, data-driven optimization. Whether by deriving mathematical bounds, building parameter-free tensor architectures, or dynamically navigating manifolds, the goal remains the same: allowing machine learning models to autonomously discover the intrinsic scale of the data they are trying to understand.