Data Uniformity Improves Training Efficiency and More, with a Convergence Framework Beyond the NTK Regime

Yuqing Wang, Shangding GuSubmission to ICLR 2026 · Rejected

TL;DR

Selecting training data to maximize data uniformity—specifically by maximizing the minimum pairwise distance between data points ($h_{\min}$)—significantly accelerates training and reduces neural network approximation error. Backed by a convergence framework that moves beyond the restrictive Neural Tangent Kernel (NTK) regime, the authors show that uniform subsets half the size of full datasets can train more than $2\times$ faster while matching or outperforming full-data performance on LLM benchmarks.

What Problem Does This Paper Address?

Curating training data is central to scaling and fine-tuning large language models (LLMs). While heuristics targeting "quality" and "diversity" are common, data selection remains mostly empirical, heuristic-driven, and task-specific.

Moreover, deep learning optimization theory often relies on the Neural Tangent Kernel (NTK) regime, which assumes near-infinite model width and infinitesimal learning rates where models essentially behave like linear approximations without true feature learning. Existing optimization frameworks also struggle to explain the non-convex training dynamics of modern architectures like transformers without unrealistic Lipschitz smoothness assumptions.

This paper bridges the gap with a quantitative, foundational principle: data uniformity. The authors analyze how the geometric spacing between training points directly dictates optimization speed and approximation error.

Methods

The authors connect geometry, optimization, and approximation through three core components:

  1. Theoretical Convergence Beyond NTK:
  2. Formulates polynomial generalized smoothness and local relaxed dissipativity, bypassing the unrealistic assumption of global Lipschitz smoothness.
  3. Utilizes tools from measure theory and differential topology (specifically the parametric transversality theorem) to prove gradient descent (GD) convergence for non-linear, multi-layer networks (including transformers with GELU, SiLU, and softmax).
  4. Proves that residual connections structurally restore and preserve the full rank of layerwise Jacobians across training.
  5. Impact of Pairwise Distance ($h_{\min}$):
  6. Establishes that higher distribution uniformity directly translates to a larger minimum pairwise distance $h_{\min}$.
  7. Proves that when points cluster too closely together (small $h_{\min}$), the lower frame bound of the gradients collapses, drastically slowing optimization.
  8. Uses computational geometry (Delaunay triangulation) and an adaptive Bramble-Hilbert lemma to prove that larger $h_{\min}$ reduces the network's approximation error over Sobolev spaces.
  9. Greedy Max-Min Data Selection Algorithm:
  10. Embeds data points using sentence or token representations (e.g., Word2Vec).
  11. Starting from a seed point, the algorithm iteratively selects the candidate sample that maximizes the minimum distance (Euclidean or cosine) to the already selected subset.

Datasets

The authors evaluated their framework across regression and language modeling tasks:

  • California Housing: Tabular regression used to validate $\ell_2$-loss SGD dynamics against coreset selection baselines.
  • TeaMs-RL (9k): An instruction-tuning dataset synthesized via reinforcement learning.
  • WizardLM (20k): An instruction-following dataset generated via Evol-Instruct.
  • LESS (14k): A gradient-influence preselected instruction dataset targeted for MMLU.
  • Evaluation Benchmarks: Evaluated using the LM Evaluation Harness on ARC Challenge, TruthfulQA MC, and MMLU.

No new benchmarks were created; instead, the authors applied distance-maximization pruning to established distillation and instruction datasets.

Key Insights and Ideas

  • Redundant Data Actively Hurts Optimization: When data points are too close together in representation space, the gradient vectors become nearly linearly dependent. This shrinks the effective gradient lower bound and stalls gradient descent.
  • Uniformity as a Proxy for More Data: According to the paper's Bramble-Hilbert error bounds, increasing $h_{\min}$ via uniform selection improves function approximation just as effectively as reducing the maximum interpolation gap $h_{\max}$ by collecting vastly more data.
  • Why Residual Connections Work: The transversality analysis reveals that non-degeneracy (full-rank Jacobians) is generic in analytic functions. Adding a shortcut/residual connection guarantees that layer mappings avoid rank collapse, providing a clean topological explanation for their practical success.
  • Encoder Invariance: Ablations show that simple, lightweight embeddings (like Word2Vec) perform just as well for distance-based selection as heavier models (like Sentence-BERT), indicating that the spatial uniformity principle matters far more than the specific embedding geometry.

Key Results

  • $2\times+$ Faster Wall-Clock Convergence:
  • On LLaMA-1-13B fine-tuned on WizardLM, the 10k uniform subset reached a loss threshold of 0.31 in 148.2 minutes, compared to 281.2 minutes for a 10k random subset and 561.6 minutes for the full 20k dataset.
  • Superior Downstream Generalization:
  • LLaMA-1-13B (WizardLM): The 10k uniform subset attained 43.82% overall accuracy (56.31% on ARC Challenge, 31.33% on TruthfulQA MC), outperforming the 10k random subset (43.09%) and matching/beating the full 20k dataset (43.63%).
  • LLaMA-1-7B (LESS): Pruning to a 7k uniform subset achieved 39.61% MMLU accuracy, outperforming the 7k random baseline (37.12%) and the full 14k dataset (38.57%).
  • Training Stability: On California Housing, uniform subsets eliminated gradient instabilities and loss spikes observed when training on the full or random subsets.

Limitations and Future Work

  • Distance Metric Discrepancy: The authors observed that selecting subsets via cosine distance yielded faster training loss decay, whereas Euclidean distance produced better downstream generalization on WizardLM. A formal theoretical explanation for this metric trade-off remains open.
  • Computational Overhead: The greedy max-min selection runs in quadratic time with respect to corpus size unless approximate nearest-neighbor search is integrated, making selection on billion-token pre-training corpora expensive.
  • Scope of Application: The experimental evaluation focuses entirely on supervised fine-tuning (SFT) and regression; validating this uniformity principle at scale during LLM pre-training is left for future exploration.

Summary updated

Abstract

Data selection plays a crucial role in data-driven decision-making, including in large language models (LLMs), and is typically task-dependent. Properties such as data quality and diversity have been extensively studied and are known to enhance model performance. However, it remains unclear whether there exist other quantitative and general principles of data selection that can consistently improve performance, especially for complicated tasks. In this paper, we demonstrate that selecting more uniformly distributed data can improve training efficiency while enhancing performance. Specifically, we establish that more uniform (less biased) distribution leads to a larger minimum pairwise distance between data points, denoted by $h_{\min}$, and prove that a smaller $h_{\min}$ can slow down the training dynamics of gradient descent (GD). Moreover, we theoretically show that the approximation error of neural networks decreases as $h_{\min}$ increases. Our analysis introduces a convergence framework for GD beyond the Neural Tangent Kernel (NTK) regime, applicable to a broad class of architectures, including transformers, without requiring Lipschitz smoothness. This framework further provides theoretical justification for the use of residual connection and function composition in deep neural architectures. In the end, we conduct comprehensive experiments for supervised fine-tuning across various settings, including different optimization strategies, model sizes, and training datasets. The results consistently demonstrate that selecting data by maximizing pairwise distance significantly accelerates training and achieves comparable or better performance in LLMs across diverse datasets. Code and Datasets are available at the link: https://anonymous.4open.science/r/data-uniformity-1A5C.
BibTeX
@misc{wang2025data,
  author = {Yuqing Wang and Shangding Gu},
  title = {{Data Uniformity Improves Training Efficiency and More, with a Convergence Framework Beyond the NTK Regime}},
  year = {2025},
  eprint = {2506.24120},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2506.24120},
}
Figure