Will Large-scale Generative Models Corrupt Future Datasets?

Ryuichiro Hataya, Han Bao, Hiromi Arai

Introduction

The rapid advancement of deep learning has historically relied on a "virtuous cycle": scraping massive amounts of human-generated data from the internet to train increasingly capable models. However, the explosion of text-to-image tools like Stable Diffusion and Midjourney has fundamentally altered this landscape. The internet is no longer a purely human-generated resource; it is increasingly populated by AI-generated content. This paper investigates the looming crisis of Dataset Contamination: the phenomenon where future models are trained on data "polluted" by the outputs of their predecessors.

The cycle of dataset contamination: generative models produce content that is scraped back into training sets for future models.

Methodology

To empirically study the impact of this contamination, the authors simulated a future internet environment through a series of controlled experiments. They generated two primary synthetic datasets using Stable Diffusion: SD-ImageNet (based on ImageNet categories) and SD-COCO (based on MS-COCO captions).

By replacing varying percentages (20%, 40%, and 80%) of original real-world images with these synthetic counterparts, the researchers created "mixed" datasets. They then trained standard architectures, including ResNet-50 and Swin Transformers, from scratch to evaluate performance across three core domains: image classification, image captioning, and image generation. Additionally, they explored whether self-supervised learning techniques, such as Masked Autoencoders (MAE), could serve as a defense against the negative effects of synthetic data.

Examples of synthetic images and captions from the SD-COCO dataset used to simulate contamination.

The Diversity Deficit and Semantic Ambiguity

The study identifies two primary drivers of model degradation: mode collapse and semantic ambiguity.

  1. The Diversity Deficit: While synthetic images often appear realistic, they lack the structural diversity found in the real world. A generative model might produce a high-quality image of an "African Elephant," but it tends to repeat specific poses and backgrounds, missing the "long tail" of rare lighting conditions and unique environments present in human-captured photography. This lack of variety limits the model's ability to generalize.
  2. Semantic Ambiguity: Generative models frequently struggle with polysemy. For instance, the ImageNet class "Titi" refers to a species of monkey, yet Stable Diffusion often generates images of people because "Titi" is also a common human name. This introduces categorical noise that human-curated datasets typically avoid.

Key Findings

The empirical results demonstrate a clear decline in performance as contamination levels increase:

  • Classification Accuracy: On ImageNet, a ResNet-50 trained on 80% contaminated data saw accuracy drop from 75.7% to 65.3%. Performance on "hard" datasets like ImageNet-A (adversarial examples) and ImageNet-R (artistic renditions) suffered even more significantly, indicating a loss of robustness.
  • Captioning Quality: Using 80% synthetic data in the SD-COCO set caused the CIDEr score to drop from 1.335 to 1.279, showing that the linguistic nuances of captions are also degraded.
  • Generation Fidelity: When a diffusion model (IDDPM) was trained on 80% contaminated data, its "Recall" (a measure of data mode coverage) dropped from 0.646 to 0.490. This suggests that models trained on AI-generated data eventually lose the capacity to produce diverse content.

Mitigation and Future Outlook

Interestingly, the researchers found that models trained on "noisy" real-world images from the web (e.g., WebVision) often outperformed those trained on "clean" synthetic images. This suggests that the inherent messiness of the real world provides more valuable training signals than polished but repetitive AI outputs.

One promising remedy is the use of self-supervised pre-training (MAE), which proved significantly more robust to contamination than standard supervised learning. However, the authors emphasize that the most effective long-term solution may be policy-driven: they advocate for the implementation of invisible watermarks by generative model providers. This would allow future researchers to filter out synthetic data, preserving the integrity of the datasets that fuel AI progress.

Summary updated

Abstract

Recently proposed large-scale text-to-image generative models such as DALL•E 2 [47], Midjourney [42], and StableDiffusion [51] can generate high-quality and realistic images from users’ prompts. Not limited to the research community, ordinary Internet users enjoy these generative models, and consequently, a tremendous amount of generated images have been shared on the Internet. Meanwhile, today’s success of deep learning in the computer vision field owes a lot to images collected from the Internet. These trends lead us to a research question: "will such generated images impact the quality of future datasets and the performance of computer vision models positively or negatively?" This paper empirically answers this question by simulating contamination. Namely, we generate ImageNet-scale and COCO-scale datasets using a state-of-the-art generative model and evaluate models trained with "contaminated" datasets on various tasks, including image classification and image generation. Throughout experiments, we conclude that generated images negatively affect downstream performance, while the significance depends on tasks and the amount of generated images. The generated datasets and the codes for experiments will be publicly released for future research. Generated datasets and source codes are available from https://github.com/moskomule/dataset-contamination.
BibTeX
@inproceedings{hataya2022will,
  author = {Ryuichiro Hataya and Han Bao and Hiromi Arai},
  title = {{Will Large-scale Generative Models Corrupt Future Datasets?}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2022},
}
Figure