Abstract
For certain infinitely-wide neural networks, the neural tangent kernel (NTK) theory fully characterizes generalization, but for the networks used in practice, the empirical NTK only provides a rough first-order approximation. Still, a growing body of work keeps leveraging this approximation to successfully analyze important deep learning phenomena and design algorithms for new applications. In our work, we provide strong empirical evidence to determine the practical validity of such approximation by conducting a systematic comparison of the behavior of different neural networks and their linear approximations on different tasks. We show that the linear approximations can indeed rank the learning complexity of certain tasks for neural networks, even when they achieve very different performances. However, in contrast to what was previously reported, we discover that neural networks do not always perform better than their kernel approximations, and reveal that the performance gap heavily depends on architecture, dataset size and training task. We discover that networks overfit to these tasks mostly due to the evolution of their kernel during training, thus, revealing a new type of implicit bias.
Introduction
The Neural Tangent Kernel (NTK) framework has emerged as a powerful tool in deep learning theory, offering a way to analyze neural networks as linear models. By taking a first-order Taylor expansion around initial weights, the NTK describes the network's evolution during training. While this approximation becomes exact in the infinite-width limit, practical networks are finite, leading to a "non-linear advantage" where real networks often outperform their kernel counterparts. This study investigates the practical utility of the NTK: when can it reliably predict generalization, and what drives the performance gap between linear and non-linear regimes?
Methodology: Probing the Linear Approximation
To systematically compare standard neural networks (MLP, LeNet, ResNet18) with their linearized versions, we utilize the Jacobian at initialization to train linear models. A key innovation in our approach is the use of NTK Eigenfunctions to define tasks. By labeling data using the binarized eigenfunctions of the NTK, we create a structured array of predictive tasks with mathematically defined complexity levels based on their corresponding eigenvalues.
As shown in Figure 1, the linearized model stays within the tangent plane $\mathcal{H}$, while the actual neural network evolves along the manifold $\mathcal{F}$. We track this evolution by measuring "kernel rotation"—the change in the empirical NTK during training—and how it affects the concentration of target label energy onto the kernel's principal components.
Empirical Findings: Ranking and the Non-Linear Disadvantage
Our experiments reveal that the NTK is an excellent "thermometer" for task difficulty. Across all tested architectures, both the neural network and its linearized version show a consistent drop in validation accuracy as the task complexity (eigenfunction index) increases. This suggests that the NTK correctly ranks which tasks are inherently harder for the network to learn.
Surprisingly, we identify a "non-linear disadvantage." As seen in Figure 2, for high-index eigenfunctions (more complex tasks), the linearized model often generalizes better than the full neural network. For instance, on specific high-index tasks, a linearized model might achieve ~93% accuracy while the full network struggles at ~65%. This contradicts the common assumption that non-linearity always provides a performance boost.
The Role of Kernel Evolution and Implicit Bias
The performance gap is largely driven by "kernel rotation." In non-linear networks, the kernel is not static; it adapts to align itself with the target task. While this adaptation allows for rapid convergence on training data, it can lead to aggressive overfitting on complex tasks that a static linear model would simply ignore.
This movement represents a significant form of implicit bias. In a ResNet18 trained on a binary CIFAR-10 task, we observed the target function "energy" captured by the top 50 kernel eigenvectors jump from ~23% at initialization to over 96% by the end of training. This alignment enables the network to fit the data efficiently but creates a trade-off: the very mechanism that allows deep networks to learn quickly can also cause them to lose generalization ability on tasks that do not align with their evolving inductive bias.
Conclusion
While the NTK framework provides a robust ranking of task complexity, its ability to predict absolute generalization is limited by the dynamic nature of finite networks. The discovery of the "non-linear disadvantage" and the mechanism of kernel rotation highlights a fundamental trade-off in deep learning. Future work should focus on controlling this rotation to harness the speed of non-linear learning without sacrificing the stability and generalization found in kernel approximations.
Summary updated