Abstract
This paper demonstrates that the theoretical "infinite-width" limit of neural networks—specifically the Neural Tangent Kernel (NTK)—provides a powerful, unified signal for improving model training before it even begins. By leveraging this training-free signal, the authors achieve state-of-the-art results in architecture selection, label cleaning, and bias detection at a fraction of the computational cost of traditional methods.
Introduction
Training modern, overparameterized deep learning models is a massive drain on time and electricity. To make training more efficient, researchers often use "helper" techniques like Neural Architecture Search (NAS) or data pruning. However, most of these techniques are "piggyback" methods—they require actually starting the training process to evaluate performance. This creates a circular problem where significant compute is spent just to figure out how to save compute. The authors address this by asking: Can we predict a model's training behavior and data quality without ever running a single gradient descent step?
Methodology
The researchers turn to Neural Tangent Kernel (NTK) theory. The NTK describes how a network's outputs change in response to its parameters during training. While the NTK of a finite network changes as it learns, the NTK of an "infinitely wide" network remains constant.
The authors propose using the Infinite-Width Gram-Matrix ($H^\infty$) as a universal diagnostic tool. They develop four specific applications for this signal:
- Architecture Selection: Using Kernel PCA on $H^\infty$ to determine if an architecture is likely to generalize well based on data clustering.
- Pseudo-Label Verification: Introducing the Infinite-Width Block Diagonalization Error ($L(Z)$) to measure how well labels fit the NTK's underlying geometry.
- Bias Identification: Analyzing off-diagonal elements of the Gram-Label matrix to identify "entangled" classes likely to be confused.
- Label Refurbishment: An iterative algorithm using the $L(Z)$ metric gradient to correct noisy or incorrect labels.
Experimental Setup
The approach was validated using standard benchmarks (D-MNIST, Fashion-MNIST, CIFAR-10, CIFAR-100) and custom synthetic datasets:
- Corners: 4 classes defined by a single bright pixel in a corner.
- Shapes: 4 classes defined by geometric patterns.
These synthetic sets proved that the NTK signal correctly identifies the optimal architecture type (e.g., Linear vs. CNN) for specific visual tasks.
Key Insights
- Unified Signal: Architecture search and label cleaning are shown to be two sides of the same coin: the alignment of data structure with network initialization.
- Efficiency: Computing properties of a theoretical "infinite" network is analytically faster than iterative training of finite ones.
- Spectral Importance: The "tail" of the NTK spectrum (low-magnitude eigenvalues) contains crucial information for generalization on difficult cases.
Key Results
- Speed: Computing the $H^\infty$ signal is 100x to 400x faster than training. For CIFAR-100, the signal is ready in 25 seconds, compared to over 15 hours for training.
- Architecture Ranking: The training-free method achieved a Ranked Biased Overlap (RBO) score of 0.829, nearly perfectly predicting post-training performance.
- Label Noise: The $L(Z)$ metric successfully ranked labeling quality across all datasets without ground-truth labels.
- Refurbishment: The algorithm matched or exceeded state-of-the-art training-based methods like BARE while being significantly more efficient.
Conclusion
The authors note that label refurbishment optimization can be delicate due to local minima. Future work includes improving optimization robustness and extending these training-free diagnostics to complex architectures like Transformers or State Space Models (SSMs).
Summary updated