Why do neural networks with millions of parameters converge so quickly on real-world data, yet struggle to fit random noise? The answer lies in a phenomenon called spectral alignment.
Recent research suggests that the training dynamics of a neural network are governed by the relationship between the target labels and the internal geometric structure of the model—specifically, the eigenvectors of its Neural Tangent Kernel (NTK) or its Gram matrix. When the "signal" in the data aligns with the most dominant "modes" of the network, learning is nearly instantaneous.
The Network as a Spectral Filter
To understand why some things are easier to learn than others, we can view a neural network through the lens of functional approximation. As Lili Su and Pengkun Yang demonstrate in On Learning Over-parameterized Neural Networks: A Functional Approximation Perspective, the speed of learning is determined by how well the target function aligns with the principal eigenfunctions of the data distribution's integral operator.
Think of the network's internal state as a collection of "learning directions" (eigenvectors). Each direction has a corresponding "speed" (eigenvalue).
- Large Eigenvalues: These are the "highways" of learning. If the target labels fall along these directions, the loss drops precipitously in the first few iterations.
- Small Eigenvalues: These are the "narrow paths." Learning here is slow and requires many updates.
Sanjeev Arora and Simon S. Du formalize this in Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks. They show that real-world datasets aren't random; their labels naturally align with the eigenvectors corresponding to the largest eigenvalues of the system's Gram matrix. This explains the "efficiency gap": real data travels the highway, while random noise is forced to take the narrow paths.
Why Alignment Leads to Generalization
This spectral perspective also solves the mystery of generalization. If a network is large enough to memorize anything, why does it choose to learn meaningful patterns first?
Samet Oymak and Mahdi Soltanolkotabi argue in Generalization Guarantees for Neural Networks via Harnessing the Low-rank Structure of the Jacobian that the network's Jacobian matrix has a low-rank structure. Meaningful data features reside in a high-magnitude "information space," while noise is relegated to a "nuisance space."
Because the information space corresponds to larger singular values, the network learns the "signal" much faster than the "noise." By the time the network would start memorizing noise, early stopping has usually already ended the training. This "Representation Alignment," as explored by Ehsan Imani and Martha White in Representation Alignment in Neural Networks, suggests that hidden layers actively reorganize to align their strongest internal structures with the target labels.
Engineering Alignment
If alignment is the key to speed, can we force it to happen? Several researchers have developed methods to do exactly that:
- Initialization: Fabian Paischer and Lukas Hauzenberger introduced Explained Variance Adaptation (EVA) in One Initialization to Rule them All. By using SVD on activations to find the directions of maximum variance, they initialize low-rank adapters (LoRA) to align with the leading subspace of the NTK, maximizing the initial gradient signal.
- Graph Structure: In Graph Neural Networks, Saurabh Sihag and Alejandro Ribeiro found that using cross-covariance graphs—rather than simple input correlation—maximizes the alignment between the NTK and the target labels, leading to faster convergence (Neural Tangent Kernels Motivate Graph Neural Networks with Cross-Covariance Graphs).
- Multi-Task Balancing: Xiaohan Qin and Junchi Yan use the maximum eigenvalue of task-specific NTKs to detect task imbalance. In NTKMTL, they show that tasks with larger eigenvalues "hog" the optimization process, and balancing these spectral properties is essential for fair multi-task learning.
The Physical Limit of Learning
How efficient can this process be? Alexander A Alemi and colleagues suggest in Speed Limits for Deep Learning that neural networks operating in the NTK regime are remarkably close to the theoretical physical limits of information processing. By analyzing the spectral decomposition of labels against the NTK, they show that training is most efficient when the target signal is distributed across the kernel's spectral components in a way that avoids "bottlenecks" in the low-eigenvalue regions.
In summary, the "black box" of neural network training is increasingly being understood as a transparent spectral process. We don't just want more parameters; we want parameters that create a kernel whose dominant eigenvectors point exactly where our labels want to go.