Abstract
Traditional data selection for Large Language Model (LLM) fine-tuning often relies on qualitative heuristics. This paper introduces a quantitative principle based on data uniformity, defined by the minimum pairwise distance ($h_{min}$) between data points in an embedding space. We present a novel convergence framework that moves beyond the restrictive Neural Tangent Kernel (NTK) regime, utilizing differential topology to model the training dynamics of complex Transformer architectures. Our findings demonstrate that maximizing $h_{min}$ leads to faster training and improved approximation error. Empirically, uniform data subsets of 10k samples match or exceed the performance of 20k full datasets while reducing training time by over 50%.
Introduction
While data quality and diversity are recognized as critical for LLM performance, the geometric distribution of data—specifically its uniformity—has remained under-explored. This research posits that "clumping" in the data manifold creates optimization redundancies that slow down gradient descent. By ensuring data is spread uniformly, we can optimize the training process for both speed and accuracy.
Theoretical Framework: Beyond the NTK Regime
Most existing theories on neural network convergence rely on the NTK regime, which assumes networks behave linearly as their width increases. However, modern Transformers, characterized by residual connections and layer normalization, deviate significantly from this linear behavior.
The authors develop a new mathematical framework for Gradient Descent (GD) that avoids the "linearization" of NTK. By applying tools from differential topology and measure theory, they prove that a larger $h_{min}$ directly correlates with faster training dynamics and lower approximation errors. This framework also provides theoretical justification for residual connections, showing they are essential for preventing the network's mapping from collapsing into lower-dimensional subspaces, thereby preserving model expressivity.
Methodology: Greedy Distance Maximization
To achieve data uniformity, the authors propose a selection strategy that maximizes the minimum pairwise distance $h_{min}$.
- Embedding: Data points are projected into an embedding space (e.g., using Word2Vec).
- Selection: A greedy algorithm iteratively selects the next data point furthest from the existing subset, ensuring the resulting distribution is as uniform as possible across the manifold.
Experimental Results
The approach was validated using LLaMA-1 (7B and 13B) models on datasets such as WizardLM and TeaMs-RL. Performance was evaluated across the ARC Challenge, TruthfulQA MC, and MMLU benchmarks.
Training Efficiency
The most significant finding was the dramatic reduction in training time. On a single A100 GPU, a 10k uniform subset of WizardLM reached target loss thresholds more than twice as fast as a 10k random subset or the full 20k dataset.
Model Performance
Despite the 50% reduction in data volume, the uniform subsets maintained or improved upon the accuracy of the full dataset. This suggests that the redundant data in the full set may actually hinder the optimization process rather than help it.
Discussion and Limitations
The study highlights that uniformity is a powerful proxy for data efficiency. However, the effectiveness of the method is sensitive to the choice of distance metric and the quality of the initial embeddings. While Cosine distance favored training speed, Euclidean distance showed potential for better generalization in some contexts. Future work will investigate whether these principles of uniformity extend from Supervised Fine-Tuning (SFT) to the massive-scale pre-training phase.
Conclusion
By shifting the focus from purely qualitative data selection to a geometric principle of uniformity, this work provides a scalable method for improving LLM training efficiency. The accompanying theoretical framework offers a more realistic model for understanding the convergence of modern deep learning architectures.
Implementation available at: https://github.com/SafeRL-Lab/data-uniformity
Summary updated