Scalable Data Valuation for Machine Learning

This direction is split into other directions: computational scalability of data valuation, computational complexity of data valuation. Its last version stays readable and citable here.

Version 1 of 1237 papers280 conceptsfrozen 2026-10-04

In the era of massive datasets, the prevailing wisdom has been that "more data is better." However, as datasets scale to billions of tokens or petabytes of sensor logs, we encounter a diminishing return: web-scraped data is often noisy, mislabeled, or redundant. This has birthed a critical research direction: Data Valuation. The goal is to assign a precise "price tag" or utility score to individual training examples to identify which points drive performance and which are merely dead weight.

The Gold Standard: Data Shapley

To value data fairly, researchers have turned to cooperative game theory. Amirata Ghorbani and James Y. Zou introduced Data Shapley: Equitable Valuation of Data for Machine Learning, which treats a machine learning model as a "game" and the training points as "players."

The intuition is simple: the value of a data point is its average marginal contribution across all possible subsets of the dataset. If a point consistently improves a model's accuracy when added to various groups of other data, it is highly valuable. This framework is uniquely "fair" because it satisfies mathematical axioms—for instance, two identical data points receive the same value, and a "null" point that adds nothing receives zero.

The Exponential Bottleneck

While mathematically elegant, exact Shapley valuation is computationally impossible for modern ML. To calculate it perfectly for $N$ points, you would theoretically need to retrain your model $2^N$ times. As Jiachen T. Wang and Ruoxi Jia note in A Note on "Towards Efficient Data Valuation Based on the Shapley Value", this exponential growth makes exact valuation a non-starter for even modest datasets.

The core of recent research is finding ways to bypass this "retraining wall" without losing the fairness of the Shapley framework.

Strategies for Scalability

Researchers have developed several distinct "shortcuts" to make data valuation practical:

1. Proxy Models and KNNs One of the most effective ways to speed up valuation is to replace the complex neural network with a simpler "proxy." Fan Wu and colleagues demonstrated in Scalability vs. Utility: Do We Have to Sacrifice One for the Other in Data Importance Quantification? that K-Nearest Neighbor (KNN) models allow for closed-form Shapley calculations. By using pre-trained embeddings, one can value data orders of magnitude faster than retraining. Dimitrios Dimitriadis and Sanmi Koyejo further optimized this in Invariant Aggregator for Defending against Federated Backdoor Attacks, reducing complexity from $O(N^K)$ to $O(N^{1.5})$.

2. Model-Agnostic Statistical Metrics Instead of measuring how data affects a specific model, some researchers look at the geometry of the data itself. Jiongli Zhu and Babak Salimi proposed KAIROS: Scalable Model-Agnostic Data Valuation, which uses Maximum Mean Discrepancy (MMD) to rank data quality. Similarly, Wenqian Li introduced FedBary, which uses Wasserstein distances to value data in federated learning without ever training a model, achieving linear complexity.

3. Meta-Learning and Reinforcement Learning Rather than calculating values post-hoc, we can learn them during training. Jinsung Yoon and Sercan O. Arik developed Data Valuation using Reinforcement Learning (DVRL), where a "data value predictor" is trained alongside the main model. The predictor receives a reward based on how much the selected data improves performance on a small, clean validation set.

Why Valuation Matters: From Pruning to Pricing

The ability to value data isn't just a theoretical exercise; it enables several downstream applications:

The Future: Scaling Laws for Value

As we move toward billion-scale datasets, the focus is shifting from "how much is this point worth now?" to "how will its value change as the dataset grows?" Ian Connick Covert has begun mapping Scaling Laws for the Value of Individual Data Points in Machine Learning, discovering predictable "value decay" curves. This suggests that the future of data-centric AI lies in understanding the lifecycle of data value, allowing us to build better models not by hoarding data, but by curating it with surgical precision.

Papers in this version

Showing 60 of 237 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_scalable_data_valuation_for_machine_lear_72837f0c,
  title = {Scalable Data Valuation for Machine Learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/scalable-data-valuation-for-machine-learning/rdr_72837f0cacf94afab4404943d70831bf},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_scalable_data_valuation_for_machine_lear_72837f0c_v1,
  title = {Scalable Data Valuation for Machine Learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/scalable-data-valuation-for-machine-learning/rdr_72837f0cacf94afab4404943d70831bf/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}