In the pursuit of efficient machine learning, the challenge has shifted from simply acquiring more data to selecting the most informative subset. While early strategies often relied on static metrics—such as prioritizing "hard" examples or ensuring "diversity"—recent research reveals that the optimal selection strategy is not fixed. Instead, it is a moving target that must adapt to the total available budget.
The Representativeness-Difficulty Tradeoff
The core intuition behind modern data selection is a dynamic shift between two poles: representativeness (prototypical, "easy" samples) and informativeness (complex, "hard" samples).
When the labeling or training budget is extremely small, models lack the foundational patterns necessary to build a coherent feature space. In these "sample-deficient" regimes, Hoyong Choi and Nohyun Ki demonstrate in BWS: Best Window Selection Based on Sample Scores for Data Pruning across Broad Ranges that prioritizing easy samples is essential for establishing basic features. Selecting hard samples too early can actually hinder performance, as the model lacks the context to interpret complex decision boundaries.
As the budget increases, the utility of these prototypical samples diminishes because they become redundant. The strategy must then pivot toward "hard" samples—those near decision boundaries or with high loss—to refine the model's accuracy and capture fine-grained details. Abhinab Acharya and Dayou Yu formalize this in Balancing Feature Similarity and Label Variability for Optimal Size-Aware One-shot Subset Selection, showing that the optimal sample profile changes fundamentally as the target subset size grows.
Quality vs. Diversity at Scale
A similar dynamic exists between data quality (how "good" or clean a sample is) and diversity (how well the subset covers the entire distribution).
In low-data regimes, quality is the primary driver of performance. A few high-quality, representative examples are more valuable than a diverse but noisy set. However, as the budget expands, diversity becomes the dominant factor. Minghao Wu and Gholamreza Haffari argue in The Best of Both Worlds: Bridging Quality and Diversity in Data Selection with Bipartite Graph that without a mechanism to prioritize diversity in larger budgets, models suffer from diminishing returns due to redundant information.
Adaptive Selection Mechanisms
To address these shifting requirements, researchers are moving away from fixed selection criteria toward adaptive, budget-dependent frameworks:
- Sigmoid-based Weighting: Zhijing Wan and colleagues propose using sigmoid functions to automatically balance representative and difficult samples based on the sampling rate in Foundation Model Insights and a Multi-Model Approach for Superior Fine-Grained One-shot Subset Selection.
- Process-Aware Objectives: In the context of In-Context Learning, Wei Zheng and Zijie Wang introduce PaCS-ICL, which uses a budget-dependent balancing factor to shift from "typical" examples to "long-tail" diversity as more demonstrations are allowed.
- Dataset Condensation: Even in synthetic data generation, the "capacity" of the distilled set dictates the source data's utility. Zekai Li and Ziyao Guo show in Prioritize Alignment in Dataset Distillation that matching the difficulty of pruned source data to the target compression ratio is key to achieving "lossless" distillation.
By treating the data budget as a primary variable rather than a constraint, these methods ensure that the selected subset provides the maximum possible information for the model's current stage of learning.