The Surprising Effectiveness of Infinite-Width NTKs for Characterizing and Improving Model Training

Joshua DeOliveira, Walter Gerych, Elke A. Rundensteiner

Abstract

This paper demonstrates that the theoretical "infinite-width" limit of neural networks—specifically the Neural Tangent Kernel (NTK)—provides a powerful, unified signal for improving model training before it even begins. By leveraging this training-free signal, the authors achieve state-of-the-art results in architecture selection, label cleaning, and bias detection at a fraction of the computational cost of traditional methods.

Introduction

Training modern, overparameterized deep learning models is a massive drain on time and electricity. To make training more efficient, researchers often use "helper" techniques like Neural Architecture Search (NAS) or data pruning. However, most of these techniques are "piggyback" methods—they require actually starting the training process to evaluate performance. This creates a circular problem where significant compute is spent just to figure out how to save compute. The authors address this by asking: Can we predict a model's training behavior and data quality without ever running a single gradient descent step?

Methodology

The researchers turn to Neural Tangent Kernel (NTK) theory. The NTK describes how a network's outputs change in response to its parameters during training. While the NTK of a finite network changes as it learns, the NTK of an "infinitely wide" network remains constant.

The authors propose using the Infinite-Width Gram-Matrix ($H^\infty$) as a universal diagnostic tool. They develop four specific applications for this signal:

  1. Architecture Selection: Using Kernel PCA on $H^\infty$ to determine if an architecture is likely to generalize well based on data clustering.
  2. Pseudo-Label Verification: Introducing the Infinite-Width Block Diagonalization Error ($L(Z)$) to measure how well labels fit the NTK's underlying geometry.
  3. Bias Identification: Analyzing off-diagonal elements of the Gram-Label matrix to identify "entangled" classes likely to be confused.
  4. Label Refurbishment: An iterative algorithm using the $L(Z)$ metric gradient to correct noisy or incorrect labels.
A comparison between existing training-heavy solutions and the proposed training-free unified approach using NTK signals.

Experimental Setup

The approach was validated using standard benchmarks (D-MNIST, Fashion-MNIST, CIFAR-10, CIFAR-100) and custom synthetic datasets:

  • Corners: 4 classes defined by a single bright pixel in a corner.
  • Shapes: 4 classes defined by geometric patterns.

These synthetic sets proved that the NTK signal correctly identifies the optimal architecture type (e.g., Linear vs. CNN) for specific visual tasks.

Key Insights

  • Unified Signal: Architecture search and label cleaning are shown to be two sides of the same coin: the alignment of data structure with network initialization.
  • Efficiency: Computing properties of a theoretical "infinite" network is analytically faster than iterative training of finite ones.
  • Spectral Importance: The "tail" of the NTK spectrum (low-magnitude eigenvalues) contains crucial information for generalization on difficult cases.

Key Results

  • Speed: Computing the $H^\infty$ signal is 100x to 400x faster than training. For CIFAR-100, the signal is ready in 25 seconds, compared to over 15 hours for training.
  • Architecture Ranking: The training-free method achieved a Ranked Biased Overlap (RBO) score of 0.829, nearly perfectly predicting post-training performance.
  • Label Noise: The $L(Z)$ metric successfully ranked labeling quality across all datasets without ground-truth labels.
  • Refurbishment: The algorithm matched or exceeded state-of-the-art training-based methods like BARE while being significantly more efficient.
Comparison of architecture rankings between the proposed training-free method and exhaustive training.

Conclusion

The authors note that label refurbishment optimization can be delicate due to local minima. Future work includes improving optimization robustness and extending these training-free diagnostics to complex architectures like Transformers or State Space Models (SSMs).

Summary updated

Abstract

Developments in deep neural nets have trended towards increasingly larger overparameterized architectures, resulting in lengthy training sessions with ever more elusive training dynamics. Thus, ensuring these models learn accurate generalizable representations of data efficiently is challenging. Previous works have developed specialized techniques from data-pruning, architecture selection, pseudo-label generation, bias identification, or label refurbishment to improve downstream training. Problematically, most methods require prohibitively expensive iterative model training. In this paper, we demonstrate that we can exploit the recent neural tangent kernel (NTK) theory for understanding and improving model training behavior before ever training a model. First, we show a powerful signal derived from the NTK theory can be computed remarkably fast. We then leverage this signal for the design of a unified suite of surprisingly effective tools for the four important tasks of architecture selection, pseudo-label verification, bias identification, and label refurbishment, all requiring zero model training.
BibTeX
@inproceedings{deoliveira2025surprising,
  author = {Joshua DeOliveira and Walter Gerych and Elke A. Rundensteiner},
  title = {{The Surprising Effectiveness of Infinite-Width NTKs for Characterizing and Improving Model Training}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i15.33786},
}
Figure