Abstract
State-of-the-art text-to-image diffusion models can produce impressive visuals but may memorize and reproduce training images, creating copyright and privacy risks. Existing prompt perturbations applied at inference time, such as random token insertion or embedding noise, may lower copying but often harm image-prompt alignment and overall fidelity. To address this, we introduce two complementary methods. First, Region-Aware Prompt Augmentation (RAPTA) uses an object detector to find salient regions and turn them into semantically grounded prompt variants, which are randomly sampled during training to increase diversity while maintaining semantic alignment. Second, Attention-Driven Multimodal Copy Detection (ADMCD) aggregates local patch, global semantic, and texture cues with a lightweight transformer to produce a fused representation, and applies simple thresholded decision rules to detect copying without training with large annotated datasets. Experiments show that RAPTA reduces overfitting while maintaining high synthesis quality, and that ADMCD reliably detects copying, outperforming single-modal metrics.
Introduction
Text-to-image diffusion models like Stable Diffusion have revolutionized content creation, yet their tendency to memorize specific training examples remains a significant hurdle. Because these models are often trained on massive, weakly curated datasets, they can over-anchor on specific image-caption pairs, leading them to reproduce training data nearly verbatim when prompted. This "memory problem" raises critical ethical, privacy, and copyright concerns.
Existing mitigation strategies often rely on inference-time perturbations, which can degrade image quality or prompt adherence. This research proposes a two-pronged approach: addressing memorization during the training phase via RAPTA and providing a robust post-training "plagiarism checker" via ADMCD. Together, these modules aim to reduce the occurrence of memorization and provide reliable tools for detecting it when it happens.
Methodology
Region-Aware Prompt Augmentation (RAPTA)
RAPTA addresses the root cause of memorization—over-anchoring on static captions—by diversifying the training prompts. Instead of a single caption (e.g., "A photo of a beach"), RAPTA employs an object detector to identify salient regions and their spatial positions within the image.
As shown in the pipeline above, the process involves three stages:
- Extract Salient Regions: A multi-scale region proposal identifies objects and their bounding boxes.
- Position Discretization: Objects are mapped to a 3x3 grid (e.g., "middle center," "top left").
- Scoring and Sampling: Semantically grounded variants (e.g., "A photo of a beach featuring a person in the middle center") are generated and weighted by semantic consistency. By randomly sampling these variants during training, the model learns generalized relationships between text and visual concepts rather than memorizing specific pairs.
Attention-Driven Multimodal Copy Detection (ADMCD)
To detect instances where a model has reproduced training data, we introduce ADMCD. Unlike standard metrics like SSIM or LPIPS, which are easily fooled by simple image corruptions or rotations, ADMCD uses a lightweight Transformer to fuse three distinct feature streams: local patches (for geometry), global semantics (via CLIP), and texture descriptors (for artistic style).
This multimodal approach allows the system to remain robust even if one "sense" is obscured by noise or editing. The system employs a two-stage thresholding logic to not only detect a copy but also classify it as either a Retrieve Copy (near-exact) or a Style Copy (mimicry of artistic patterns).
Experimental Evaluation
Datasets and Robustness
Performance was evaluated using LAION-10k to measure general copy rates. For detection accuracy, a curated set of 1,200 query-reference pairs was used, including exact copies, style copies, and non-copy pairs. Robustness was further tested against ImageNet-C, subjecting the detector to various "attacks" such as heavy blur, salt-and-pepper noise, and 30° rotations.
Key Results
- Reduced Overfitting: RAPTA reduced the absolute copy rate by up to 64.9% compared to baseline Stable Diffusion 2.1 models.
- Maintained Fidelity: Visual quality, measured by FID and KID scores, remained high, indicating that the diversity introduced by RAPTA does not compromise the model's creative capabilities.
- Superior Detection: ADMCD significantly outperformed baselines like SSCD and DreamSim. While other metrics' scores plummeted under image corruption, ADMCD remained stable, providing a clear gap between true copies and similar-looking images.
Discussion and Conclusion
This work demonstrates that memorization is not an inevitable byproduct of diffusion training. By forcing the model to generalize through semantically grounded prompt diversity (RAPTA), we can significantly lower the risk of verbatim reproduction. Furthermore, the multimodal nature of ADMCD provides a more nuanced and resilient tool for copyright protection than single-modal metrics.
Limitations: RAPTA's effectiveness is currently tied to the accuracy of the underlying object detector (e.g., Faster R-CNN). Additionally, scaling ADMCD to reference galleries containing billions of images presents an ongoing engineering challenge. Future work will focus on optimizing these modules for massive-scale deployment and exploring detector-agnostic augmentation strategies.
Summary updated