In machine learning, the prevailing wisdom was once "more data is better." However, as datasets scale to trillions of tokens or billions of images, the focus has shifted toward efficiency. The challenge is no longer just acquiring data, but selecting the right subset.
Traditional data selection often relies on pointwise metrics—scoring each example in isolation based on its "difficulty" or similarity to a target. But this approach has a fundamental flaw: it ignores the collective structure of the data. Selecting the top 10% of "most similar" sentences often results in a redundant, skewed subset that lacks the diversity of the original domain.
Instead, a growing body of research proposes distributional alignment. Rather than asking "how good is this individual point?", these methods ask "how well does this entire subset represent the shape of the target distribution?"
The Pointwise Selection Trap
When we select data points one by one based on high similarity scores, we often over-select "easy" or high-frequency examples. Sethy et al. observed this in language modeling, where standard filtering over-selected common phrases, losing the "tail" of the distribution necessary for accuracy. Similarly, Wu et al. found that for speech recognition, a mix of "hard" and "easy" examples is required to mirror the statistical profile of the task.
Distributional alignment solves this by treating the subset as a single entity. The goal is to minimize the distance between the probability distribution of the selected subset and a target distribution (such as a small, high-quality validation set or the entire unlabeled pool).
Optimal Transport and the Wasserstein Distance
A powerful tool for this alignment is Optimal Transport (OT). OT provides a way to measure the "cost" of transforming one distribution into another. By minimizing the Wasserstein distance, researchers can select a subset that globally matches the target's geometry.
Guzman-Nateras et al. utilized the Sinkhorn algorithm to find the best alignment between source and target data, allowing them to integrate semantic similarity and task relevance into a single optimization. This global view is crucial; as Feng et al. argue in their work on TAROT, traditional methods assume the influence of data points is additive. In reality, data influence is complex and multimodal. By framing selection as an OT problem, the authors preserve the structural diversity of the target distribution, which is particularly effective for complex tasks like motion prediction.
This geometric approach also enables better performance prediction. Kang et al. showed that the OT distance between a training mixture and a validation set serves as a learning-agnostic proxy for model accuracy, allowing practitioners to select optimal data mixtures without training full-scale models.
Kernel Methods and Maximum Mean Discrepancy
Another common way to measure distributional distance is Maximum Mean Discrepancy (MMD). MMD measures the distance between the means of two distributions in a high-dimensional feature space (a Reproducing Kernel Hilbert Space).
This is the engine behind prototype selection. Kim et al. introduced MMD-critic to identify "prototypes" that statistically approximate a dataset, while also identifying "criticisms"—points that the prototypes fail to explain. Dhurandhar et al. extended this with ProtoDash, which assigns importance weights to these prototypes, allowing for a more flexible and efficient representation of the underlying data.
In active learning, MMD ensures that as we query new labels, the labeled set doesn't drift away from the overall data population. Tang et al. and Hwang et al. both used MMD to ensure that selected samples reflect the global data density, preventing the model from focusing too narrowly on uncertain boundaries or outliers.
Greedy Matching and Diversity
How do we actually pick the points to satisfy these distributional goals? Many researchers employ greedy iterative selection.
Acharya et al. proposed matching the subset mean to the dataset's geometric median. Their algorithm picks a point, calculates the residual error between the current subset mean and the target, and then picks the next point to best cancel out that error. This "herding" behavior naturally promotes diversity: if the current subset is over-represented in one area of the feature space, the algorithm is mathematically forced to look elsewhere for the next sample.
Why Distributional Alignment Works
The primary advantage of these methods is robustness. By focusing on the global distribution rather than individual points, these methods naturally filter out noise. Xiao et al. demonstrated that minimizing the Wasserstein distance to the global dataset prioritizes central, representative samples over noisy outliers, making the resulting "coreset" more robust to corrupted data.
Furthermore, it allows for extreme data efficiency. Karbasi et al. showed that by selecting just 1% of a dataset using OT-based alignment, one can outperform models trained on the entire repository. Whether it is matching neuron activations to improve testing efficiency (Li et al.), aligning latent quantiles for dataset condensation (Wei et al.), or selecting synthetic data that mirrors real-world distributions (Zeng et al.), the core intuition remains the same: the best training set is the one that most faithfully recreates the shape of the problem you are trying to solve.
Key Foundational Work
An Active Learning Approach to Task Adaptation (2011) — contributed 1 concept elements to this direction.
Corpus design based on the kullback-leibler divergence for text-to-speech synthesis application (2006) — contributed 1 concept elements to this direction.
Text data acquisition for domain-specific language models (2006) — contributed 1 concept elements to this direction.