Modern neural networks are defined by millions, or even billions, of parameters. Yet, a growing body of research suggests that the actual "work" of learning occurs within a tiny fraction of this high-dimensional space. By restricting optimization to low-dimensional subspaces, researchers have uncovered fundamental properties of model complexity and developed algorithms that are significantly faster and more memory-efficient than standard gradient descent.
The Intrinsic Dimension of Learning
If a model has a million parameters, does it truly need a million degrees of freedom to solve a task? Li et al. investigated this by forcing neural networks to train within randomly oriented, low-dimensional subspaces. They discovered that for many common benchmarks, models could reach 90% of their full-parameter performance using a subspace that was orders of magnitude smaller than the original parameter space. This "intrinsic dimension" serves as a measure of a task's complexity, revealing that the solution manifolds for deep learning are remarkably low-dimensional.
This phenomenon isn't just a curiosity; it follows predictable geometric patterns. Larsen et al. demonstrated that a "phase transition" occurs during training: once the subspace dimension exceeds a specific threshold determined by the loss landscape's geometry, the probability of finding a successful solution jumps from near-zero to near-certainty. Even extreme compression is possible; Abbasi showed that high-performing models can be represented as simple linear combinations of frozen, random basis networks, effectively reducing the trainable parameters to a small set of mixture coefficients.
Accelerating Convergence with Curvature Subspaces
While random subspaces help us measure model capacity, intelligently chosen subspaces can drastically speed up optimization. The primary challenge in large-scale optimization is that second-order methods (which use curvature information to take smarter steps) are computationally ruinous in high dimensions because they require calculating or storing the Hessian matrix.
Subspace methods bypass this by restricting the second-order search to a small, relevant slice of the parameter space:
- Krylov Subspaces: Povey proposed constructing a subspace from the gradient and recent update directions. Optimizing directly within this Krylov subspace allows for faster convergence than standard Hessian-free methods by avoiding complex damping heuristics.
- Gradient History: Methods like those discussed by Scieur and Lee et al. restrict updates to a subspace spanned by a limited history of gradients. This ensures the cost per iteration scales linearly with the number of variables while still capturing essential curvature information.
- Adaptive Subspaces: The Sum of Functions Optimizer (SFO) by Sohl‐Dickstein et al. maintains independent Hessian approximations for minibatches within a shared, adaptive subspace, providing the benefits of quasi-Newton methods without the prohibitive memory overhead.
Efficiency through Geometric Restriction
Beyond speed, subspace constraints make otherwise impossible problems tractable. In Riemannian optimization, maintaining constraints like orthogonality (e.g., ensuring $X^\top X = I$) requires expensive "retraction" steps. Han et al. showed that by restricting updates to low-dimensional random submanifolds, the computational bottleneck shifts from geometry-preserving operations back to simple gradient computations, enabling faster wall-clock convergence.
Similarly, in online learning, Jézéquel et al. demonstrated that complex non-linear regularization can be performed efficiently by solving the optimization problem within the span of the current feature vector and the cumulative gradient. This makes the complexity per round independent of the total number of samples.
Why Subspaces Work: Invariant Dynamics
The success of these methods stems from the fact that learning dynamics are naturally "compressible." Yaras et al. and Kwon et al. argue that in overparameterized models, weight updates tend to occur within invariant, low-dimensional subspaces. By identifying these subspaces early—often based on initial gradient directions—we can filter out the noise of redundant dimensions.
This insight has practical implications for model adaptation and compression. For instance, Chen et al. showed that learned optimizers, which are usually too memory-intensive for large models, become viable when their updates are constrained to a tiny intrinsic subspace. By focusing computational resources only on the dimensions that actually drive learning, we can achieve the performance of massive architectures at a fraction of the cost.
Key Foundational Work
Manifold Identification of Dual Averaging Methods for Regularized Stochastic Online Learning (2011) — contributed 2 concept elements to this direction.
Krylov Subspace Descent for Deep Learning (2012) — contributed 1 concept elements to this direction.