Budget-Aware Data Selection Dynamics

This direction is split into other directions: scale-dependent data optimization. Its last version stays readable and citable here.

Version 1 of 126 papers36 conceptsfrozen 2026-10-04

In the pursuit of efficient machine learning, the challenge has shifted from simply acquiring more data to selecting the most informative subset. While early strategies often relied on static metrics—such as prioritizing "hard" examples or ensuring "diversity"—recent research reveals that the optimal selection strategy is not fixed. Instead, it is a moving target that must adapt to the total available budget.

The Representativeness-Difficulty Tradeoff

The core intuition behind modern data selection is a dynamic shift between two poles: representativeness (prototypical, "easy" samples) and informativeness (complex, "hard" samples).

When the labeling or training budget is extremely small, models lack the foundational patterns necessary to build a coherent feature space. In these "sample-deficient" regimes, Hoyong Choi and Nohyun Ki demonstrate in BWS: Best Window Selection Based on Sample Scores for Data Pruning across Broad Ranges that prioritizing easy samples is essential for establishing basic features. Selecting hard samples too early can actually hinder performance, as the model lacks the context to interpret complex decision boundaries.

As the budget increases, the utility of these prototypical samples diminishes because they become redundant. The strategy must then pivot toward "hard" samples—those near decision boundaries or with high loss—to refine the model's accuracy and capture fine-grained details. Abhinab Acharya and Dayou Yu formalize this in Balancing Feature Similarity and Label Variability for Optimal Size-Aware One-shot Subset Selection, showing that the optimal sample profile changes fundamentally as the target subset size grows.

Quality vs. Diversity at Scale

A similar dynamic exists between data quality (how "good" or clean a sample is) and diversity (how well the subset covers the entire distribution).

In low-data regimes, quality is the primary driver of performance. A few high-quality, representative examples are more valuable than a diverse but noisy set. However, as the budget expands, diversity becomes the dominant factor. Minghao Wu and Gholamreza Haffari argue in The Best of Both Worlds: Bridging Quality and Diversity in Data Selection with Bipartite Graph that without a mechanism to prioritize diversity in larger budgets, models suffer from diminishing returns due to redundant information.

Adaptive Selection Mechanisms

To address these shifting requirements, researchers are moving away from fixed selection criteria toward adaptive, budget-dependent frameworks:

By treating the data budget as a primary variable rather than a constraint, these methods ensure that the selected subset provides the maximum possible information for the model's current stage of learning.

Papers in this version

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_budget_aware_data_selection_dynamics_7108fa31,
  title = {Budget-Aware Data Selection Dynamics},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/budget-aware-data-selection-dynamics/rdr_7108fa31659b4d2fad49ff9d358cc8e4},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_budget_aware_data_selection_dynamics_7108fa31_v1,
  title = {Budget-Aware Data Selection Dynamics},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/budget-aware-data-selection-dynamics/rdr_7108fa31659b4d2fad49ff9d358cc8e4/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}