Data Uniformity Improves Training Efficiency and More, with a Convergence Framework Beyond the NTK Regime

Yuqing Wang, Shangding Gu

Abstract

Traditional data selection for Large Language Model (LLM) fine-tuning often relies on qualitative heuristics. This paper introduces a quantitative principle based on data uniformity, defined by the minimum pairwise distance ($h_{min}$) between data points in an embedding space. We present a novel convergence framework that moves beyond the restrictive Neural Tangent Kernel (NTK) regime, utilizing differential topology to model the training dynamics of complex Transformer architectures. Our findings demonstrate that maximizing $h_{min}$ leads to faster training and improved approximation error. Empirically, uniform data subsets of 10k samples match or exceed the performance of 20k full datasets while reducing training time by over 50%.

Introduction

While data quality and diversity are recognized as critical for LLM performance, the geometric distribution of data—specifically its uniformity—has remained under-explored. This research posits that "clumping" in the data manifold creates optimization redundancies that slow down gradient descent. By ensuring data is spread uniformly, we can optimize the training process for both speed and accuracy.

Theoretical Framework: Beyond the NTK Regime

Most existing theories on neural network convergence rely on the NTK regime, which assumes networks behave linearly as their width increases. However, modern Transformers, characterized by residual connections and layer normalization, deviate significantly from this linear behavior.

The authors develop a new mathematical framework for Gradient Descent (GD) that avoids the "linearization" of NTK. By applying tools from differential topology and measure theory, they prove that a larger $h_{min}$ directly correlates with faster training dynamics and lower approximation errors. This framework also provides theoretical justification for residual connections, showing they are essential for preventing the network's mapping from collapsing into lower-dimensional subspaces, thereby preserving model expressivity.

Methodology: Greedy Distance Maximization

To achieve data uniformity, the authors propose a selection strategy that maximizes the minimum pairwise distance $h_{min}$.

  1. Embedding: Data points are projected into an embedding space (e.g., using Word2Vec).
  2. Selection: A greedy algorithm iteratively selects the next data point furthest from the existing subset, ensuring the resulting distribution is as uniform as possible across the manifold.

Experimental Results

The approach was validated using LLaMA-1 (7B and 13B) models on datasets such as WizardLM and TeaMs-RL. Performance was evaluated across the ARC Challenge, TruthfulQA MC, and MMLU benchmarks.

Training Efficiency

The most significant finding was the dramatic reduction in training time. On a single A100 GPU, a 10k uniform subset of WizardLM reached target loss thresholds more than twice as fast as a 10k random subset or the full 20k dataset.

Training time comparison across different loss thresholds.

Model Performance

Despite the 50% reduction in data volume, the uniform subsets maintained or improved upon the accuracy of the full dataset. This suggests that the redundant data in the full set may actually hinder the optimization process rather than help it.

Performance comparison on ARC Challenge and TruthfulQA MC benchmarks.

Discussion and Limitations

The study highlights that uniformity is a powerful proxy for data efficiency. However, the effectiveness of the method is sensitive to the choice of distance metric and the quality of the initial embeddings. While Cosine distance favored training speed, Euclidean distance showed potential for better generalization in some contexts. Future work will investigate whether these principles of uniformity extend from Supervised Fine-Tuning (SFT) to the massive-scale pre-training phase.

Conclusion

By shifting the focus from purely qualitative data selection to a geometric principle of uniformity, this work provides a scalable method for improving LLM training efficiency. The accompanying theoretical framework offers a more realistic model for understanding the convergence of modern deep learning architectures.

Implementation available at: https://github.com/SafeRL-Lab/data-uniformity

Summary updated

Abstract

Data selection plays a crucial role in data-driven decision-making, including in large language models (LLMs), and is typically task-dependent. Properties such as data quality and diversity have been extensively studied and are known to enhance model performance. However, it remains unclear whether there exist other quantitative and general principles of data selection that can consistently improve performance, especially for complicated tasks. In this paper, we demonstrate that selecting more uniformly distributed data can improve training efficiency while enhancing performance. Specifically, we establish that more uniform (less biased) distribution leads to a larger minimum pairwise distance between data points, denoted by $h_{\min}$, and prove that a smaller $h_{\min}$ can slow down the training dynamics of gradient descent (GD). Moreover, we theoretically show that the approximation error of neural networks decreases as $h_{\min}$ increases. Our analysis introduces a convergence framework for GD beyond the Neural Tangent Kernel (NTK) regime, applicable to a broad class of architectures, including transformers, without requiring Lipschitz smoothness. This framework further provides theoretical justification for the use of residual connection and function composition in deep neural architectures. In the end, we conduct comprehensive experiments for supervised fine-tuning across various settings, including different optimization strategies, model sizes, and training datasets. The results consistently demonstrate that selecting data by maximizing pairwise distance significantly accelerates training and achieves comparable or better performance in LLMs across diverse datasets. Code and Datasets are available at the link: https://anonymous.4open.science/r/data-uniformity-1A5C.
BibTeX
@misc{wang2025data,
  author = {Yuqing Wang and Shangding Gu},
  title = {{Data Uniformity Improves Training Efficiency and More, with a Convergence Framework Beyond the NTK Regime}},
  year = {2025},
  eprint = {2506.24120},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2506.24120},
}
Figure