For years, the dominant paradigm in training large language models (LLMs) has been "more is better." Early scaling laws suggested that performance scales predictably with dataset size and parameter count. However, this brute-force scaling introduces a hidden tax: noise, hallucinations, and conflicting signals.
When a model is trained on massive, unvetted web crawls, it encounters contradictory facts, poorly formatted text, and low-quality reasoning steps. During optimization, these noisy examples introduce conflicting gradients. Instead of converging toward a clean representation of logic or language, the model's parameters must accommodate a fractured optimization landscape. This dilution of signal explains why massive models can still struggle with reasoning or hallucinate basic facts.
By shifting the focus from data volume to data integrity, a growing body of research demonstrates that smaller, meticulously curated datasets can achieve or exceed the performance of models trained on a massive "data tsunami."
Filtering the Data Tsunami in Instruction Tuning
Instruction tuning is particularly sensitive to data quality. If an instruction-tuning dataset contains thousands of repetitive, low-effort, or contradictory examples, the model learns to generate generic or hallucinated responses rather than precise answers.
To quantify this, Yulei Qin et al. in Unleashing the Power of Data Tsunami: A Comprehensive Survey on Data Assessment and Selection for Instruction Tuning of Language Models showed that selecting a highly effective subset—often as small as 5% of the original corpus—frequently yields superior performance compared to training on the entire dataset.
Why does a fraction of the data work better? Jiaxi Yang et al. explored this mechanism in One-Shot Learning as Instruction Data Prospector for Large Language Models, demonstrating that training on just the top 1% of a dataset avoids the conflicting signals that degrade model performance during brute-force scaling. By removing the long tail of redundant and noisy data, the model's optimization path remains focused on high-fidelity instruction-following patterns.
Furthermore, data quality is not just about cleanliness; it is also about difficulty and boundary exploration. Instead of passive filtering, Yuxin Jiang et al. introduced an adversarial distillation framework in Lion: Adversarial Distillation of Proprietary Large Language Models. By using a teacher LLM to identify and generate "hard" instructions tailored specifically to a student model's current weaknesses, the authors proved that high-performance reasoning can be unlocked with relatively small datasets. The intuition here is akin to active learning: a model learns more by confronting a small set of its own mistakes than by reviewing millions of examples it already understands.
Domain Alignment and Specialized Curation
When adapting models to highly specialized domains—such as law, materials science, or remote sensing—the limitations of massive, general-purpose datasets become even more pronounced. In these fields, general web text acts as background noise that dilutes the precise, domain-specific terminology and structures required for expert tasks.
The authors of A Systematic Study of the Role of Data Quality and Alignment for Fine-tuning LLMs for Enhanced Autoformalization (Krrish Chawla et al.) argue that data alignment—how closely the training data matches the target task—is far more critical than sheer data quantity. Higher alignment directly translates to lower perplexity and better downstream performance.
This principle enables smaller, domain-specific models to punch far above their weight:
- Legal Text: Stella Douka et al. developed JuriBERT: A Masked-Language Model Adaptation for French Legal Text, demonstrating that pre-training a smaller architecture from scratch on a compact, high-quality legal corpus outperforms massive generic models trained on vastly larger datasets.
- Recommendation Systems: In Could Small Language Models Serve as Recommenders? Towards Data-centric Cold-start Recommendation, Xuansheng Wu et al. showed that selecting a small volume of highly relevant documents for pre-training allows an 11-million parameter model to match the performance of models 30 times its size.
- Multimodal Remote Sensing: Yiguo He et al. leveraged advanced LLMs to generate a curated image-text dataset in Enhancing Remote Sensing Vision-Language Models Through MLLM and LLM-Based High-Quality Image-Text Dataset Generation, achieving state-of-the-art performance with substantially less data.
- Scientific Discovery: For highly technical fields like carbon capture, Eryu Wang et al. demonstrated in A Dataset for Investigations of Amine-Impregnated Solid Adsorbent for Direct Air Capture that a small, manually curated dataset of just 1,336 data points extracted from literature is sufficient to train highly accurate predictive models.
The underlying mechanism is straightforward: by isolating the model from irrelevant distributions, every parameter update directly reinforces the specialized knowledge graph required for the task.
Overcoming Low-Resource Bottlenecks Through Integrity
In low-resource language settings, the "more data" approach is fundamentally blocked by a lack of digital text. However, attempting to solve this by aggressively scraping every available web page often backfires, introducing severe lexical inconsistencies and translation noise.
Instead, prioritizing data integrity allows models to thrive despite limited scale. Akintunde Oladipo et al. demonstrated this in Better Quality Pre-training Data and T5 Models for African Languages. By meticulously auditing existing web crawls and performing targeted crawls of credible sources, the authors built a high-quality corpus that allowed smaller models to double the cross-lingual effectiveness of much larger models trained on unverified web text.
Similarly, when dealing with non-standardized scripts or dialects, linguistic normalization acts as a force multiplier. In Lexical Variation and Sentiment Analysis of Roman Urdu Sentences with Deep Neural Networks, Muhammad Arslan Manzoor et al. showed that unifying spelling variations in Roman Urdu significantly outweighs raw dataset volume. When the model does not have to waste parameter capacity learning dozens of spelling variations for the same word, it can allocate its capacity to learning semantic relationships. As summarized by Damith Premasiri et al. in the Proceedings of the First Workshop on Language Models for Low-Resource Languages, rigorous data curation provides a viable, sustainable path for AI development in resource-constrained linguistic environments.
Corpus Composition and Emergent Capabilities
The benefits of data quality extend beyond task-specific fine-tuning into the foundational capabilities of models, such as in-context learning (ICL) and semantic representation.
In a study on pretraining composition, Seongjin Shin (On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Model) revealed that ICL performance is highly sensitive to the source domain and inherent diversity of the training set rather than just raw volume. A smaller, well-balanced corpus can unlock few-shot capabilities that a massive, homogeneous dataset cannot, challenging the assumption that low perplexity on a massive corpus automatically translates to few-shot intelligence.
This is supported across several linguistic and task-specific domains:
- Abstractive Summarization: Yang Janet Liu et al. showed in GUMSum: Multi-Genre Data and Evaluation for English Abstractive Summarization that fine-tuning on a tiny amount of expert-vetted, multi-genre data yields better abstractive quality and fewer hallucinations than relying on massive zero-shot models.
- Question Answering: Changwook Jun et al. (ANNA: Enhanced Language Representation for Question Answering) demonstrated that forcing models to learn contextual richness from clean, structured noun phrases yields better extractive answering performance than unvetted web crawls.
- Semantic Nuance and Drift: For capturing subtle semantic relationships or linguistic drift, Elizaveta Kuzmenko et al. (Clustering of Russian Adjective-Noun Constructions using Word Embeddings) and Tatsuya Aoki et al. (Distinguishing Japanese Non-standard Usages from Standard Ones) both found that corpus balance and representativeness outweigh sheer volume. Flooding a model with uncurated text degrades its ability to detect subtle linguistic patterns.
- Language Recognition: Even in speech and language recognition, Hui Zhao et al. (Results of The 2015 NIST Language Recognition Evaluation) confirmed that limited, highly relevant training data can match or exceed the efficacy of massive, mismatched datasets.
Ultimately, high-quality, heterogeneous training corpora—as demonstrated by Hang Le et al. with FlauBERT : des modèles de langue contextualisés pré-entraînés pour le français—enable models to build robust representation spaces with significantly less data. When the training data is diverse, clean, and balanced, the model spends its parameter budget learning the underlying structure of the task rather than memorizing noise.
Key Foundational Work
The study of the effect of training set on statistical language modeling (2001) — contributed 2 concept elements to this direction.
Approaches to Improving Corpus Quality for Statistical Machine Translation (2011) — contributed 1 concept elements to this direction.