In the era of massive datasets, the prevailing wisdom has been that "more data is better." However, as datasets scale to billions of tokens or petabytes of sensor logs, we encounter a diminishing return: web-scraped data is often noisy, mislabeled, or redundant. This has birthed a critical research direction: Data Valuation. The goal is to assign a precise "price tag" or utility score to individual training examples to identify which points drive performance and which are merely dead weight.
The Gold Standard: Data Shapley
To value data fairly, researchers have turned to cooperative game theory. Amirata Ghorbani and James Y. Zou introduced Data Shapley: Equitable Valuation of Data for Machine Learning, which treats a machine learning model as a "game" and the training points as "players."
The intuition is simple: the value of a data point is its average marginal contribution across all possible subsets of the dataset. If a point consistently improves a model's accuracy when added to various groups of other data, it is highly valuable. This framework is uniquely "fair" because it satisfies mathematical axioms—for instance, two identical data points receive the same value, and a "null" point that adds nothing receives zero.
The Exponential Bottleneck
While mathematically elegant, exact Shapley valuation is computationally impossible for modern ML. To calculate it perfectly for $N$ points, you would theoretically need to retrain your model $2^N$ times. As Jiachen T. Wang and Ruoxi Jia note in A Note on "Towards Efficient Data Valuation Based on the Shapley Value", this exponential growth makes exact valuation a non-starter for even modest datasets.
The core of recent research is finding ways to bypass this "retraining wall" without losing the fairness of the Shapley framework.
Strategies for Scalability
Researchers have developed several distinct "shortcuts" to make data valuation practical:
1. Proxy Models and KNNs One of the most effective ways to speed up valuation is to replace the complex neural network with a simpler "proxy." Fan Wu and colleagues demonstrated in Scalability vs. Utility: Do We Have to Sacrifice One for the Other in Data Importance Quantification? that K-Nearest Neighbor (KNN) models allow for closed-form Shapley calculations. By using pre-trained embeddings, one can value data orders of magnitude faster than retraining. Dimitrios Dimitriadis and Sanmi Koyejo further optimized this in Invariant Aggregator for Defending against Federated Backdoor Attacks, reducing complexity from $O(N^K)$ to $O(N^{1.5})$.
2. Model-Agnostic Statistical Metrics Instead of measuring how data affects a specific model, some researchers look at the geometry of the data itself. Jiongli Zhu and Babak Salimi proposed KAIROS: Scalable Model-Agnostic Data Valuation, which uses Maximum Mean Discrepancy (MMD) to rank data quality. Similarly, Wenqian Li introduced FedBary, which uses Wasserstein distances to value data in federated learning without ever training a model, achieving linear complexity.
3. Meta-Learning and Reinforcement Learning Rather than calculating values post-hoc, we can learn them during training. Jinsung Yoon and Sercan O. Arik developed Data Valuation using Reinforcement Learning (DVRL), where a "data value predictor" is trained alongside the main model. The predictor receives a reward based on how much the selected data improves performance on a small, clean validation set.
Why Valuation Matters: From Pruning to Pricing
The ability to value data isn't just a theoretical exercise; it enables several downstream applications:
- Efficient Pruning: In autonomous driving, where datasets are petabyte-scale, Zhaoyang Liu and others showed in Are All Data Necessary? that maximizing trajectory entropy can prune 40% of data while maintaining performance.
- Data Markets: As AI labs compete for high-quality human annotations, frameworks like fairshare by Luyang Zhang and Chenyan Xiong use valuation to ensure annotators are paid based on the actual utility of their work, preventing market collapse (Fairshare Data Pricing via Data Valuation for Large Language Models).
- Continual Learning: To prevent "catastrophic forgetting," Zheda Mai and Scott Sanner developed ASER, which uses Shapley values to select "sentinel" memory samples that protect old knowledge while the model learns new tasks (Adversarial Shapley Value Experience Replay for Task-Free Continual Learning).
The Future: Scaling Laws for Value
As we move toward billion-scale datasets, the focus is shifting from "how much is this point worth now?" to "how will its value change as the dataset grows?" Ian Connick Covert has begun mapping Scaling Laws for the Value of Individual Data Points in Machine Learning, discovering predictable "value decay" curves. This suggests that the future of data-centric AI lies in understanding the lifecycle of data value, allowing us to build better models not by hoarding data, but by curating it with surgical precision.