When training machine learning models, the traditional wisdom has been that more data is better, provided it is clean and diverse. Standard data pruning and curation techniques reflect this philosophy by filtering datasets for global properties: maximizing overall representativeness, ensuring class balance, or removing low-perplexity noise.
However, a growing body of research demonstrates that a dataset optimized for broad generalization is often suboptimal when a model needs to specialize. Whether deploying a model into a specific localized environment with a distinct distributional shift, or fine-tuning a large language model for a narrow domain like mathematics, global data metrics fall short.
Target-driven data selection flips the problem on its head. Instead of asking "what is the best general subset of this data?", it asks "what specific subset will maximize performance on our intended target distribution?"
The Mismatch of Global Metrics
Why do general-purpose data filters fail when we want a model to specialize? The core issue is a lack of task alignment. When a model is trained on a massive, globally diverse dataset, it allocates its parameter capacity across a wide array of features. If the deployment target requires highly specific expertise, much of that capacity is wasted on orthogonal concepts.
As Hulkund et al. demonstrate in DataS3: Dataset Subset Selection for Specialization, models trained on general datasets frequently exhibit suboptimal performance when deployed under localized distributional shifts. Standard selection practices that prioritize broad representativeness fail to prepare models for these target shifts.
This mismatch is particularly pronounced in multimodal and foundation models. In CLIPLoss and Norm-Based Data Selection Methods for Multimodal Contrastive Learning, Wang et al. show that general-purpose filtering ignores the unique feature requirements of downstream evaluation sets. Without a mechanism to anchor selection to a target distribution, even high-quality training samples can act as statistical noise relative to the desired downstream capabilities. As noted by Nikdan et al. in Efficient Data Selection at Scale via Influence Distillation, data relevance is inherently relative; without a defined target distribution to guide optimization, general-purpose pruning cannot reliably surface the specialized knowledge a model requires.
Quantifying Target Alignment via Influence
To select data for a specific target, we need a mathematical way to score how much an individual training sample will benefit that target. The most direct mechanism for this is influence estimation, which measures how a small change in a training point's weight affects the model's loss on a target validation set.
To compute this at scale, Nikdan et al. utilize second-order optimization information to calculate exactly how much a given sample reduces loss on a target task. Rather than treating data selection as a static preprocessing step, this allows data curation to become a direct proxy for target performance. Similarly, Wang et al. introduce an online selection framework in GREATS: Online Selection of High-Quality Data for LLM Training in Every Iteration. By using Taylor expansions to approximate the inner product between training gradients and target domain requirements, they dynamically select the most informative samples at every iteration of training.
[Massive General Dataset]
│
▼
[Influence / Taylor Evaluation] <─── Conditioned on ─── [Target Distribution]
│
▼
[Task-Aligned Training Subset]
│
▼
[Specialized Model]
Because calculating these second-order gradients across billions of parameters is computationally intense, recent work focuses on scaling these methods via distillation. In Neural Networks for Learnable and Scalable Influence Estimation of Instruction Fine-Tuning Data, Agarwal et al. demonstrate that a tiny proxy neural network can be trained to predict full-scale influence scores. This allows practitioners to bypass expensive influence functions entirely, cutting compute costs by up to 99% while retaining the performance gains of target-specific data selection.
Submodular and End-to-End Set Optimization
While influence functions evaluate samples independently, target-driven selection can also be framed as a collective set optimization problem. The goal here is to select a cohesive group of samples that collectively cover the target distribution without redundant overlap.
One prominent framework for this is parameterized submodular information measures. In PRISM: A Rich Class of Parameterized Submodular Information Measures for Guided Data Subset Selection, Kothawade et al. develop functions that mathematically reward similarity to a target set (such as a rare data slice or a specific domain) while penalizing redundancy within the selected subset. This ensures that the chosen data is both highly relevant to the target and internally diverse.
Alternatively, the selection mechanism itself can be learned end-to-end alongside the downstream task. Andreis et al. propose this in Set Based Stochastic Subsampling, treating data as sets rather than fixed vectors. By optimizing the selection model jointly with the downstream objective, the framework automatically discovers and extracts the precise feature structures required by the target task, outperforming general-purpose heuristics at extremely low sampling rates.
Why Subsets Can Outperform the Whole
A recurring, counterintuitive finding across target-driven data selection literature is that training on a carefully curated subset can significantly outperform training on the entire dataset.
In few-shot learning and instruction tuning, adding more data blindly introduces optimization variance and gradient conflict. As Chang et al. show in The SelectGen Challenge: Finding the Best Training Samples for Few-Shot Neural Text Generation, task-specific selection heuristics drastically reduce performance variance compared to random or broad sampling. When the training instances are tightly aligned with the target generation task, the model converges faster and generalizes better within that domain.
Furthermore, strategic pruning prevents high-quality but irrelevant data from diluting the gradient updates meant for the target task. In Stratified Selective Sampling for Instruction Tuning with Dedicated Scoring Strategy, Mirza et al. combine targeted difficulty scoring with stratified clustering. This dual approach ensures that the model is not just fed "good" data, but data that actively expands its capabilities relative to the target domain, maximizing data efficiency.
By shifting the data curation paradigm from global representation to target alignment, these methods allow models to achieve superior downstream performance with a fraction of the data, transforming data selection into a core component of model specialization.
Key Foundational Work
Set Based Stochastic Subsampling (2022) — contributed 1 concept elements to this direction.
PRISM: A Rich Class of Parameterized Submodular Information Measures for Guided Data Subset Selection (2022) — contributed 1 concept elements to this direction.
The SelectGen Challenge: Finding the Best Training Samples for Few-Shot Neural Text Generation (2021) — contributed 1 concept elements to this direction.