TL;DR
DAVINZ (Data Valuation at Initialization) is a training-free method that estimates the value of data points using Neural Tangent Kernel (NTK) theory. By analyzing a model at the exact moment of initialization, it achieves over a 30x speedup compared to traditional training-based methods while maintaining high accuracy. This breakthrough makes data valuation practical for large-scale deep neural networks.
The Bottleneck in Data Valuation
In the world of machine learning, not all data is created equal. Some samples are high-quality gold, while others are noisy, irrelevant, or even malicious. Data valuation—the process of assigning a mathematical "value" to specific datasets—is crucial for fair compensation in data marketplaces, identifying poisoned data, and selecting the best subsets for training (data summarization).
The gold standard for valuation involves metrics like the Shapley Value (SV) or Leave-One-Out (LOO). However, calculating these requires training a model to convergence for hundreds or thousands of different data combinations. For modern Deep Neural Networks (DNNs), where a single training run can take hours or days, this is computationally unaffordable. Previous attempts to speed this up often relied on oversimplified proxy models (like KNNs) or ignored the reality that training data might not perfectly match the distribution of the validation set (domain discrepancy).
The DAVINZ Approach
The authors introduce DAVINZ, a method that leverages the Neural Tangent Kernel (NTK). The NTK is a theoretical tool that describes how a neural network's outputs change during training. Crucially, in the "infinite width" limit, the NTK stays constant, allowing researchers to predict a model's final performance using only its state at initialization.
The core technical contribution of DAVINZ is a domain-aware generalization bound. Most NTK-based performance estimates assume the training and validation data come from the exact same distribution. DAVINZ breaks this assumption by incorporating a "domain discrepancy" term, calculated using Maximum Mean Discrepancy.
The valuation score for a dataset is essentially composed of two parts:
- In-domain complexity: How much "information" the model can extract from the training data.
- Out-of-domain discrepancy: How well the training data aligns with the specific preferences and distribution of the validation set.
By combining these factors, DAVINZ can assign a value to data without ever running a single step of backpropagation.
Key Insights and Trustworthy Properties
Initialization is surprisingly predictive. You don't need to see the weights after 100 epochs to know if a dataset is "good." The geometry of the loss landscape at the very start is highly indicative of the final converged performance.
Furthermore, data value is relative, not absolute. It depends entirely on the validation set. If your goal is to recognize cats, a high-quality image of a car has low value. DAVINZ captures this "data preference" mathematically through its domain-aware bound.
The paper proves that DAVINZ theoretically satisfies four essential properties for trustworthy data valuation:
- Preference Awareness: Values data more if it matches the validation task.
- Quantity Awareness: Generally assigns higher value to larger datasets, reflecting diminishing returns.
- Noise Stability: Small amounts of noise don't wildly swing the valuation.
- Model Robustness: The valuation remains consistent even if you slightly change the DNN architecture (e.g., VGG13 vs. VGG16).
Key Results
Across benchmarks including MNIST, CIFAR-10, and Tiny ImageNet, DAVINZ proved highly effective:
- Massive Speedup: DAVINZ is consistently 30x to 60x faster than training-based methods. For example, valuating a ResNet18 on CIFAR-10 took ~3.2 minutes with DAVINZ versus ~211 minutes for traditional training.
- High Correlation: Despite being training-free, DAVINZ scores show a high Pearson correlation (often $>0.9$) with the "ground truth" (the actual performance of a fully trained model).
- Scalability: The authors successfully calculated Shapley Values for a collaboration of 100 data contributors on a ResNet18—a task that would be virtually impossible using traditional training-based pipelines.
- Data Summarization: In experiments, DAVINZ could identify and remove 70% of the least valuable data from a set while maintaining nearly the same validation accuracy as the full dataset.
Limitations and Future Work
While powerful, DAVINZ has some limitations. Calculating the full NTK matrix is memory-intensive ($O(m^2)$ where $m$ is the number of samples). The authors use a "diagonal block approximation" to make this feasible, but very large-scale deployments still require careful memory management.
Additionally, while the theory is general, the experiments focused on standard CNNs and MLPs. Exploring how DAVINZ performs on Transformers or more exotic architectures remains an open area for future research. Finally, the method introduces a hyperparameter ($\kappa$) to balance in-domain vs. out-of-domain error; automated tuning could further simplify the process for end-users.
Summary updated