For years, a fundamental tension has divided machine learning theory. On one side sit Fixed Representation Methods—such as static kernel methods, random features, and neural networks operating in the "lazy" or Neural Tangent Kernel (NTK) regime. These methods map data into a high-dimensional space using a predefined, unchanging dictionary of features. While they enjoy elegant convex optimization landscapes, they suffer from a fatal flaw: they cannot learn new representations from the data.
On the other side sits Adaptive Feature Learning. When a neural network is trained via gradient descent beyond the lazy regime, it actively alters its internal weights to discover and refine its own basis functions. As Gilad Yehudai et al. explain in RedEx: Beyond Fixed Representation Methods via Convex Optimization, escaping the constraints of a fixed feature space allows models to solve complex tasks that require exponential resources for static regimes.
The Mechanism of Kernel and Basis Alignment
Why exactly does changing features during training make such a massive difference? The answer lies in alignment. Instead of measuring data similarity through a rigid, predefined lens, adaptive models warp their internal geometry to match the specific structure of the target function.
This can be understood as learning a "data-dependent kernel." In Mean-field Analysis on Two-layer Neural Networks from a Kernel Perspective, Shokichi Takakura et al. show that two-layer neural networks in the mean-field limit implicitly optimize their internal similarity metrics. By aligning this data-dependent kernel with the target function, the network can isolate low-dimensional manifolds embedded within high-dimensional spaces.
Rather than spreading representational capacity uniformly across a massive space, gradient descent concentrates the model's parameters exactly where the data lives. For instance, Shunta Akiyama et al. demonstrate in Excess Risk of Two-Layer ReLU Neural Networks in Teacher-Student Settings and its Superiority to Kernel Methods that a student network dynamically tunes its internal parameters to align with the specific basis functions of the target distribution. This active refinement allows the network to achieve significantly faster learning rates than any static linear estimator. Taiji Suzuki et al. reinforce this in Benefit of deep learning with non-convex noisy gradient descent: Provable excess risk bound and superiority to kernel methods, proving that the dynamic selection of basis functions allows deep networks to bypass the sample complexity requirements that cripple fixed kernels.
Even when working within kernel frameworks, introducing adaptivity yields substantial benefits. The authors of Residual Kernel Policy Network: Enhancing Stability and Robustness in RKHS-Based Reinforcement Learning show that a neural network can be used to transform input data so that it fits a fixed kernel, mitigating the need for manual hyperparameter tuning. Alternatively, one can parameterize the kernel itself: Hanyuan Hang et al. in Optimal Learning with Anisotropic Gaussian SVMs and Yangzheng Wu et al. in Pseudo-Keypoint RKHS Learning for Self-supervised 6DoF Pose Estimation utilize learnable weights within the kernel function to dynamically prioritize important feature dimensions, mimicking the flexibility of full representation learning.
Fixed Kernel Regime:
[Input Data] ---> [Static Kernel Mapping (Fixed Space)] ---> [Linear Predictor]
|
(Cannot change feature space)
Adaptive Feature Learning:
[Input Data] ---> [Dynamic Weights (Gradient Descent)] ---> [Aligned Data-Dependent Kernel]
| |
+<--- (Optimized to match data) <---+
Breaking the Curse of Dimensionality
The theoretical consequence of this alignment is an exponential separation in sample and computational complexity between the two regimes. When a problem possesses hidden low-dimensional or hierarchical structures, fixed kernels suffer from the curse of dimensionality, while adaptive feature learning thrives.
A classic illustration of this is the sparse parity problem. In Learning Parities with Neural Networks, Amit Daniely et al. provide a rigorous mathematical proof that linear and fixed-feature methods require exponential resources or samples to solve sparse parities. Conversely, a simple two-layer neural network trained with gradient descent adaptively discovers the non-linear structure, solving the problem in polynomial time.
This statistical-to-computational gap appears across various complex data distributions:
- Higher-Order Correlations: In Learning from higher-order correlations, efficiently: hypothesis tests, random features, and neural networks, Eszter Szekely et al. analyze the spiked cumulant model. They show that while neural networks can efficiently exploit non-Gaussian higher-order correlations using $O(d^2)$ samples, static random features fail completely in this regime.
- Hierarchical Structures: Hengyu Fu et al. demonstrate in Learning Hierarchical Polynomials of Multiple Nonlinear Features that adaptive feature learning allows networks to recover complex hierarchical polynomials efficiently. The sample complexity scales with the underlying feature structure rather than the polynomial degree of the combining function.
Distinguishing Distributions and Navigating Complex Landscapes
The power of adaptive feature learning extends beyond standard regression to tasks like distribution discrimination (the core engine of Generative Adversarial Networks) and reinforcement learning alignment.
When comparing high-dimensional probability distributions, fixed-kernel metrics are often "blind" to intricate, high-frequency statistical patterns. In Separation Results between Fixed-Kernel and Feature-Learning Probability Metrics, Carles Domingo-Enrich et al. prove that feature-learning discriminators—where both input and output layers are optimized—can easily distinguish distributions that fixed kernels fail to separate.
Expanding on this intuition, Carles Domingo-Enrich in Depth and Feature Learning are Provably Beneficial for Neural Network Discriminators shows that depth and active feature learning allow a discriminator to tune its internal weights to match the specific frequency components of a target distribution. Because fixed-feature models are trapped within a predefined Hilbert space, they cannot adapt to these patterns, creating a stark performance gap.
Finally, this adaptability proves crucial when aligning models to complex, non-smooth environments. In Inference-time Alignment with Rewards in Besov Spaces: Provable Advantages of Feature Learning and Multi-Step Policy Updates, Naoki Nishikawa et al. model reward landscapes using Besov spaces, which capture highly localized, "bumpy" reward functions. They prove that adaptive feature learning allows neural networks to dynamically concentrate their representational capacity on these local irregularities, drastically reducing regret compared to rigid, fixed-feature linear models.
Key Foundational Work
Learning Parities with Neural Networks (2020) — contributed 1 concept elements to this direction.
Optimal Learning with Anisotropic Gaussian SVMs (2018) — contributed 1 concept elements to this direction.