When training large language models (LLMs), the prevailing wisdom has long been that more data equals better performance. However, as highlighted by Bolin Zhang et al. in their comprehensive survey, instruction fine-tuning frequently operates under a "less is more" paradigm. Massive, uncurated instruction datasets are often saturated with redundant examples, low-quality filler text, and outright noise. Training on these superfluous points scales up computational costs linearly while providing diminishing returns—or worse, actively degrading the model's ability to align with human intent.
The core insight of recent research is that we can achieve identical or superior downstream performance by training on a tiny, carefully selected fraction (often 5% to 20%) of the original dataset. To do this effectively, we must answer two fundamental questions: How do we score the value of an individual instruction, and how do we ensure the selected subset covers the entire target behavior space?
Quantifying the Value of an Instruction
Not all data points contribute equally to a model's learning trajectory. Some samples are too easy, teaching the model nothing new; others are noisy or irrelevant, misguiding convergence. Researchers have developed several distinct lenses to quantify a sample's intrinsic value.
Instruction-Following Difficulty (IFD)
One intuitive approach is to target samples that the model finds challenging. Ming Li et al. introduced the concept of Instruction-Following Difficulty (IFD) in Cherry-LLM. The underlying intuition is simple: if a model can already generate a high-quality response unconditionally (without the instruction), the sample only teaches basic language modeling. However, if the response becomes significantly easier to generate only when conditioned on the instruction, the sample provides a strong alignment signal. By isolating these high-IFD samples, the authors demonstrated that training on just 5% of a dataset could outperform training on the full 100%.
Reward Maximization and Influence
Instead of looking at next-token loss or difficulty, we can evaluate data based on how much it improves the model's alignment score. Wu Yang et al. developed ROSE, a framework that selects data by explicitly maximizing expected rewards on a small preference validation set.
Similarly, we can measure a sample's influence—how much the model's parameters or outputs change in response to that specific data point. Ishika Agarwal et al. introduced DELIFT, which leverages utility metrics inspired by In-Context Learning to identify samples that are highly informative for the model's current parameter state.
Model Self-Perception and Complexity
Models can also be used to diagnose their own data needs. Yi Ren et al. proposed CPQS-Tuning, which trains a small classifier directly on the target model's internal hidden states. This allows the system to filter data based on the model's own "self-perception" of what it understands versus what it struggles with.
Alternatively, structural features of the text itself can signal value. In the context of domain-specific continual pre-training, Yi Ying et al. developed a framework called CDF-GC that prioritizes "knowledge-dense" samples by measuring their grammatical and lexical complexity, ensuring the model spends its compute budget on rich linguistic structures rather than repetitive phrasing.
Maximizing Diversity and Coverage
Scoring samples independently is only half the battle. If a selection algorithm simply picks the top 10% highest-scoring samples, it risks selecting thousands of minor variations of the exact same high-value prompt. This causes the model to overfit to a narrow slice of tasks. To prevent this, data selection must optimize for a coreset—a compressed subset that geometrically or semantically spans the entire dataset.
Gradient-Based Coresets
Text embeddings capture semantic similarity, but they do not necessarily reflect how a model learns. To bridge this gap, Peiqi Wang et al. utilized Determinantal Point Processes (DPPs) in their work to measure diversity based on model weight gradients rather than surface text. If two prompts produce nearly identical gradient updates, they are functionally redundant to the model's optimization path, meaning one can be safely discarded.
Building on this geometric intuition, Jipeng Zhang et al. developed TAGCOS, which clusters gradient features and applies a greedy selection algorithm to build a task-agnostic coreset, slashing data volume by 95% while preserving full-dataset performance.
Semantic Graphs and Submodular Optimization
Other approaches map the dataset as a complex topology. Yicheng Chen et al. introduced MIG, which treats the semantic space of instructions as a structured graph. By iteratively selecting nodes that maximize information gain across the graph, the algorithm ensures uniform coverage of the entire skill space.
[Massive Raw Dataset]
│
▼ (Clustering / Graph Construction)
[Semantic Clusters / Graph Nodes] ──► (Identify Redundancy & Information Conflicts)
│
▼ (Submodular Selection / Shapley Proxies)
[High-Diversity, High-Signal Coreset (5-10% Size)]
When assembling these subsets, we must also manage conflicting information. Powei Chang et al. designed SPICE, a framework that uses submodular optimization to explicitly penalize information conflict and redundancy, curating a balanced training set.
To make this computationally feasible at scale, frameworks like SHED by Yexiao He et al. and LSDC by Zhaoguang Long et al. first cluster the dataset into manageable partitions. Yexiao He et al. then compute Shapley values on representative "proxy" samples within those clusters, drastically reducing the combinatorial explosion of calculating data contributions.
For broader categorization, Keming Lu et al. introduced #InsTag, an open-set tagging system that explicitly counts the diversity and complexity of tags per sample. They found that a smart scalpel of 6,000 highly diverse, multi-tagged samples easily beats a blunt sledgehammer of 100,000 poorly distributed samples. This aligns with foundational instance selection frameworks like BDIS, developed by Fuyuan CAO et al. in their paper, which uses Bayesian decision theory and percolation theory to systematically separate essential instances from harmful or redundant noise.
Overcoming the Computational Overhead of Filtering
A major irony of data selection is that computing influence scores, gradients, or high-dimensional metrics for millions of samples using a massive LLM can cost more compute than simply training on the full dataset in the first place. To make data pruning practical, researchers have focused on building lightweight proxies.
Weak-to-Strong Filtering
Can a small, cheap model act as a scout for a giant model? Ming Li et al. explored this in Superfiltering, demonstrating a surprising "weak-to-strong" data transfer property. They showed that a tiny model like GPT-2 (124M parameters) can effectively rank data quality for a much larger model like LLaMA2-13B. Because fundamental data attributes—like grammatical coherence, clarity, and basic instruction structure—are perceptible even to small models, using a tiny proxy speeds up the data filtering pipeline by 20x without sacrificing selection quality.
Learnable Influence Networks
Taking the proxy concept to its logical extreme, Ishika Agarwal developed NN-CIFT, a method that trains a lightweight, dedicated neural network specifically to predict the expensive influence scores of instruction data. By bypassing the need to run full-scale influence functions across massive LLMs, this approach slashes the computational cost of data selection by up to 99%, making high-fidelity data pruning accessible for massive, multi-million sample corpora.
Token-Level Pruning
Finally, redundancy does not just exist between samples; it exists within them. Even in a high-quality instruction sample, a model wastes compute processing redundant filler words or tokens it can already predict with near-100% confidence. Yuchen Yang et al. introduced XTF, an explainable token-level noise filtering framework. Instead of dropping entire sentences, XTF evaluates individual tokens based on their reasoning importance, novelty, and task relevance, masking out uninformative tokens during the forward pass. This demonstrates that data efficiency can be optimized at the granular level of individual tokens, ensuring that every floating-point operation directly drives model alignment.
Key Foundational Work
ROSE: A Reward-Oriented Data Selection Framework for LLM Task-Specific Instruction Tuning (2024) — contributed 1 concept elements to this direction.
From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuning (2024) — contributed 1 concept elements to this direction.
Superfiltering: Weak-to-Strong Data Filtering for Fast Instruction-Tuning (2024) — contributed 1 concept elements to this direction.