Matching-based Data Valuation for Generative Model

Jiaxi Yang, Wenglong Deng, Benlin Liu, Yangsibo Huang, Xiaoxiao Li

Abstract

Data valuation is critical in machine learning, as it helps enhance model transparency and protect data properties. Existing data valuation methods have primarily focused on discriminative models, neglecting deep generative models that have recently gained considerable attention. Similar to discriminative models, there is an urgent need to assess data contributions in deep generative models as well. However, previous data valuation approaches mainly relied on discriminative model performance metrics and required model retraining. Consequently, they cannot be applied directly and efficiently to recent deep generative models, such as generative adversarial networks and diffusion models, in practice. To bridge this gap, we formulate the data valuation problem in generative models from a similarity-matching perspective. Specifically, we introduce Generative Model Valuator (GMValuator), the first model-agnostic approach for any generative models, designed to provide data valuation for generation tasks. We have conducted extensive experiments to demonstrate the effectiveness of the proposed method. To the best of their knowledge, GMValuator is the first work that offers a training-free, post-hoc data valuation strategy for deep generative models.

Introduction: The Challenge of Valuing Generative Data

In discriminative AI, such as image classifiers, valuing training data is relatively straightforward: if removing a specific sample causes a drop in accuracy, that sample is deemed valuable. However, for generative models like Generative Adversarial Networks (GANs) and Diffusion models, pinning down the "value" of a training point is significantly more complex.

Existing valuation methods suffer from three primary hurdles:

  1. Metric Instability: Common generative metrics like Fréchet Inception Distance (FID) or log-likelihood are often inconsistent and do not always align with human perception of quality.
  2. Computational Prohibitivity: Traditional approaches like Shapley Values or Influence Functions typically require retraining the model multiple times or calculating expensive Hessian matrices. This is a non-starter for massive modern architectures.
  3. Model Specificity: Most current tools are architecturally dependent, working only for specific types of models like VAEs.

GMValuator addresses these gaps by reframing the problem: Which training samples are most similar to the high-quality images our model actually produces?

The GMValuator Framework

GMValuator treats data valuation as a similarity-matching problem rather than a model-parameter problem. This allows it to be entirely training-free and model-agnostic. The pipeline consists of three core stages.

The GMValuator pipeline: from similarity matching and quality assessment to final value calculation.

1. Efficient Similarity Matching (ESM)

To handle datasets containing millions of images, the framework employs a two-phase retrieval process. In the Recall Phase, a pre-trained encoder (such as CLIP) and Product Quantization (PQ) are used to rapidly identify the "top-k" most similar training samples for every generated image. In the Re-ranking Phase, these results are refined using deep perceptual metrics like LPIPS or DreamSim, which capture human-like similarity better than raw pixel distances.

2. Image Quality Assessment (IQA)

Not all generated samples are successful; some may be noisy or distorted. If a model generates a low-quality image that happens to resemble training image A, image A should not receive high credit. GMValuator uses MANIQA to score the quality of generated samples. These scores ($q_m$) weight the contribution, ensuring only high-quality outputs pass significant value back to their training ancestors.

3. Value Calculation

The final value of a training sample is the sum of its quality-adjusted contributions across the entire set of generated data. This formulation ensures that training data which consistently inspires high-quality, diverse generations is valued most highly.

Key Insights: The Retrieval Pivot

The most significant conceptual shift in this work is the realization that for a well-trained generator, the generated distribution should closely overlap with the training distribution. Therefore, similarity in a robust feature space serves as a mathematically sound proxy for contribution.

Visualization of distribution overlap between training subsets (v1, v2) and generated data.

Because GMValuator does not rely on model gradients or retraining, it functions as a "black-box" tool. It can be applied to models where the training process is unknown or inaccessible. Furthermore, by incorporating IQA as a filter, the system avoids "false positives" where training samples are credited for low-quality, noisy outputs.

Experimental Results and Scalability

The authors evaluated GMValuator across diverse datasets, including MNIST, CIFAR-10, CelebA, and large-scale benchmarks like ImageNet (14M images). Key findings include:

  • Massive Speedups: On MNIST, GMValuator (without re-ranking) takes just 0.25 seconds to attribute a sample, compared to 47.9 seconds for VAE-TracIn. For GANs, it performed valuation in ~2,100 seconds, while the baseline IF4GAN required over 14,500 seconds.
  • Truthfulness: In tests with known ground truths (e.g., ensuring generated airplanes are attributed to training airplanes), GMValuator consistently outperformed baselines in attribution accuracy.
  • Out-of-Distribution (OOD) Detection: The tool successfully assigned lower values to noisy or OOD training samples, proving its ability to identify unhelpful or "poisoned" data.

Conclusion and Future Directions

GMValuator represents the first training-free, post-hoc data valuation strategy for deep generative models. While currently focused on vision-related tasks, the underlying logic of similarity matching is potentially applicable to Large Language Models (LLMs). Future work may explore applications in intellectual property and copyright, using GMValuator to determine how much a specific artist's work contributed to an AI-generated image for royalty purposes.

Summary updated

Abstract

Data valuation is critical in machine learning, as it helps enhance model transparency and protect data properties. Existing data valuation methods have primarily focused on discriminative models, neglecting deep generative models that have recently gained considerable attention. Similar to discriminative models, there is an urgent need to assess data contributions in deep generative models as well. However, previous data valuation approaches mainly relied on discriminative model performance metrics and required model retraining. Consequently, they cannot be applied directly and efficiently to recent deep generative models, such as generative adversarial networks and diffusion models, in practice. To bridge this gap, we formulate the data valuation problem in generative models from a similarity-matching perspective. Specifically, we introduce Generative Model Valuator (GMValuator), the first model-agnostic approach for any generative models, designed to provide data valuation for generation tasks. We have conducted extensive experiments to demonstrate the effectiveness of the proposed method. To the best of their knowledge, GMValuator is the first work that offers a training-free, post-hoc data valuation strategy for deep generative models.
BibTeX
@misc{yang2023matching,
  author = {Jiaxi Yang and Wenglong Deng and Benlin Liu and Yangsibo Huang and Xiaoxiao Li},
  title = {{Matching-based Data Valuation for Generative Model}},
  year = {2023},
  eprint = {2304.10701},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2304.10701},
}
Figure