Budget-Aware Data Selection Strategies

17 concepts11 papers1 proposal

In the quest to train more efficient models, researchers often ask: "Which data points are the most valuable?" The traditional answer involves identifying "high-quality" samples—those that are either highly representative of the task or particularly challenging for the model. However, a growing body of research suggests that the definition of "valuable" is not static. Instead, the optimal data mixture depends fundamentally on the available budget, whether measured in annotation costs, number of samples, or total compute.

The Representativeness-Diversity Shift

When a data budget is extremely tight, a model needs to capture the "gist" of the distribution. In these scenarios, Zheng et al. argue that prioritizing representativeness—selecting typical, central examples—is more effective for performance, particularly in in-context learning. By focusing on the core of the distribution, the model establishes a stable baseline.

As the budget increases, the marginal utility of these central samples drops. The focus must then shift toward diversity. Renduchintala et al. demonstrate this in the context of instruction tuning: while low budgets benefit from central task clusters, larger budgets require outliers to capture the full complexity of the distribution. Similarly, Wu et al. find that while quality drives performance for small datasets, diversity becomes the dominant factor for generalization as the sample count grows.

From Easy Prototypes to Hard Boundaries

A parallel trade-off exists between "easy" samples (prototypes) and "hard" samples (boundary cases). The intuition is that easy samples provide a reliable foundation, while hard samples refine the model's decision boundaries.

Grosz et al. show that when the pruning ratio is high (i.e., very little data is kept), models perform best when trained on easy, representative prototypes. If one were to select only hard, uncertain samples at a low budget, the model might fail to learn the basic structure of the classes. However, as the budget expands, Acharya et al. observe that the strategy should shift toward difficult samples to capture fine-grained details.

Wan et al. formalize this using a sigmoid-based weighting function that dynamically adjusts the balance between representative and informative samples based on the sampling rate. This ensures that the model first masters the "easy" concepts before being challenged by the "hard" ones.

Scaling Laws for Data Quality

The relationship between data selection and scale extends beyond the number of samples to the size of the model itself. Tirumala et al. found that the efficiency gains from sophisticated data selection scale positively with model size, meaning larger architectures are even more sensitive to data quality and redundancy.

This led to the development of "Information Scaling Laws" (InfoLaw). Liu et al. and Zhou et al. propose that pretraining should be viewed as a process of information accumulation rather than just token consumption. Their framework predicts that:

  • Small models benefit more from repeating high-quality data, as they take longer to saturate on specific information.
  • Large models act as more efficient learners that saturate quickly on repeated data, making high diversity essential to avoid performance stagnation.

Practical Implications

For practitioners, these findings suggest that "static" data curation—using the same filtering criteria regardless of whether you are training a 1B or 70B parameter model—is suboptimal. Instead, the selection criteria should be viewed as a function of the target scale.

As Shao et al. suggest, the trade-off parameters for data selection can often be derived theoretically from the budget itself, rather than through expensive empirical searches. By matching the complexity of the data to the capacity of the model and the size of the budget, we can achieve higher accuracy with significantly less compute.

Key Foundational Work

Balancing Feature Similarity and Label Variability for Optimal Size-Aware One-shot Subset Selection (2024) — contributed 3 concept elements to this direction.
Data Pruning via Separability, Integrity, and Model Uncertainty-Aware Importance Sampling (2024) — contributed 3 concept elements to this direction.
SMART: Submodular Data Mixture Strategy for Instruction Tuning (2024) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_scale_dependent_data_optimization_4bc4dc7b,
  title = {scale-dependent data optimization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/scale-dependent-data-optimization/rdr_4bc4dc7bbb784cecb5af3076dba6f559},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_scale_dependent_data_optimization_4bc4dc7b_v1,
  title = {scale-dependent data optimization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/scale-dependent-data-optimization/rdr_4bc4dc7bbb784cecb5af3076dba6f559/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake