Training modern large language models (LLMs) requires massive datasets, often spanning trillions of tokens. However, not all data is created equal; web scrapes contain noise, redundancy, and low-utility text that can degrade model performance or waste valuable compute. Ideally, we would score and filter every data point using our target model before training. But doing so introduces a paradox: evaluating massive datasets with a multi-billion parameter model is often as computationally expensive as the training run itself.
To break this bottleneck, a powerful paradigm has emerged: proxy-based data selection. The core intuition behind this approach is that data utility is largely an intrinsic property of the data and the task distribution, rather than an artifact of a model's specific size or architecture. Consequently, we can use small, computationally cheap "proxy" or "surrogate" models to score, rank, and filter data. These selection decisions transfer remarkably well to target models that are orders of magnitude larger.
The Mechanism of Cross-Scale Rank Consistency
Why should a tiny model with a fraction of the capacity of a target model be able to judge what a large model needs to learn? The answer lies in cross-scale rank consistency: small and large models share a highly correlated perception of data quality, difficulty, and informativeness.
[Raw Dataset]
│
▼
┌───────────────┐
│ Small Proxy │ ◄── Low compute cost (e.g., GPT-2 or 1M-param model)
│ Model │
└───────┬───────┘
│ (Scores data via loss, perplexity, or gradients)
▼
┌───────────────┐
│ High-Utility │ ◄── Pruned subset (e.g., top 10% of data)
│ Data Subset │
└───────┬───────┘
│ (Transfers seamlessly due to rank consistency)
▼
┌───────────────┐
│ Large Target │ ◄── High performance with up to 25x less compute
│ Model │
└───────────────┘
This consistency manifests in several distinct ways:
Perplexity and Loss Reduction
A common way to measure data utility is by evaluating how much a sample reduces a model's uncertainty. David Brandfonbrener et al. demonstrate this in CoLoR-Filter: Conditional Loss Reduction Filtering for Targeted Language Model Pre-training, where they use two small auxiliary models to compute conditional loss reduction. They show that data selected by these small proxies allows a 1.2B parameter target model to match baseline performance while using up to 25x less data.
Similarly, Yong Xie et al. show in Efficient Continual Pre-training for Building Domain Specific Large Language Models that document perplexity rankings remain highly correlated across different model scales, allowing cheap surrogate models to accurately curate domain-specific text.
Weak-to-Strong Consistency
In the context of instruction tuning, Ming Li et al. introduce the concept of "Superfiltering" in Superfiltering: Weak-to-Strong Data Filtering for Fast Instruction-Tuning. They demonstrate that a tiny 124M-parameter GPT-2 model can effectively rank the instruction-following difficulty of data for a LLaMA2-13B model. By training on just the top 5% of data selected by the weak proxy, the strong target model can outperform a baseline trained on the entire dataset.
Geometric and Embedding Invariance
Data selection can also happen without explicit language modeling metrics. Kyriakos Axiotis demonstrates in Data-Efficient Learning via Clustering-Based Sensitivity Sampling: Foundation Models and Beyond that vector representations generated by small, computationally efficient models can serve as geometric proxies. The loss patterns and sensitivities mapped out in the embedding space of the small model accurately predict the behavior of much larger architectures, enabling high-quality data pruning via clustering.
Capturing Training Dynamics and Trajectories
Beyond static scoring, researchers have found that the way a model learns a data point over time—its training trajectory—is also scale-invariant.
Instead of looking at a single snapshot of a model's loss, we can track how a sample's loss changes across training epochs. In SmallToLarge (S2L): Scalable Data Selection for Fine-tuning Large Language Models by Summarizing Training Trajectories of Small Models, Yu Yang et al. show that clustering examples based on the training loss trajectories of a tiny proxy model identifies high-value samples for massive target models. This allows fine-tuning to match full-dataset performance using as little as 11% of the data.
This concept is mirrored in vision and core deep learning tasks. Haizhong Zheng et al. introduce ELFS in ELFS: Label-Free Coreset Selection with Proxy Training Dynamics, showing that coreset selection based on the training behaviors of a small proxy model transfers seamlessly to larger, more complex target architectures. The underlying principle remains the same: if a data point is learned early or late by a small model, it follows a remarkably similar trajectory in a large model.
Optimizing Macro Data Mixtures
Proxy models are not limited to filtering individual samples; they can also optimize the macro-level "recipe" or mixture of data domains (e.g., the ratio of code, web text, and academic papers) for large-scale pre-training.
Finding the optimal data mixture has traditionally been a manual process. However, Qian Liu et al. propose RegMix in RegMix: Data Mixture as Regression for Language Model Pre-training. They train hundreds of ultra-small, 1M-parameter models on various data blends and use regression to predict the performance of a model 1,000x larger. Because the relative effectiveness of data mixtures is scale-invariant, the optimal mixture found by these tiny proxies transfers directly to 7B-parameter models, slashing the compute needed for mixture optimization by 90%.
This transferability of distribution-level importance is further supported by David Grangier et al. in Adaptive Training Distributions with Scalable Online Bilevel Optimization. The authors show that data importance weights learned via online bilevel optimization on a small proxy model can guide the training distribution of a significantly larger model, confirming that data relevance is primarily a property of the task and data distribution rather than model capacity.
Boundaries and Exceptions to Proxy Transferability
While proxy-based data selection is highly effective for general capabilities, optimization, and filtering, it faces boundaries when dealing with behaviors that only manifest at scale.
Emergent Behaviors
Certain properties of large models do not exist in small models, making them impossible to proxy. In Emergent and Predictable Memorization in Large Language Models, Stella Biderman et al. investigate whether smaller proxy models can forecast which specific text strings a large model will memorize verbatim. They discover that memorization is often an emergent property, meaning that small proxies fail to predict the exact strings a larger model will memorize.
Inverse Proxying (Teacher-Student Dynamics)
In scenarios involving complex, long-range context dependencies, the relationship can flip. Longyun Wu explores this in LongAttn: Selecting Long-context Training Data via Token-level Attention, showing that for long-context data selection, a larger, more capable teacher model is often required to score and filter data for a smaller student model. This is because the capacity to recognize intricate, distant semantic dependencies scales with model size; a small model may lack the attention resolution to judge whether a long document is actually useful.
Conclusion
Proxy-based data selection works because deep learning models, regardless of scale, share a fundamental learning order and sensitivity landscape determined by the data distribution itself. By offloading data valuation to lightweight surrogates, researchers can treat data curation as an independent, highly parallelizable preprocessing step. This decouples the computational cost of data selection from the ever-increasing scale of production models.