WIP: Auditing Artist Style Pirate in Text-to-image Generation Models

Linkang Du, Zheng Zhu, Min Chen, Shouling Ji, Peng Cheng, Jiming Chen, Zhikun Zhang2024 · DOI

Abstract

The rapid advancement of text-to-image (T2I) diffusion models has introduced significant challenges regarding intellectual property, specifically "style piracy." This occurs when models are fine-tuned on an artist's portfolio without authorization to replicate their unique aesthetic. We present StyleAuditor, a post-hoc auditing framework that treats an artist's style as an intrinsic "fingerprint" extracted from multi-layered neural network features. StyleAuditor identifies infringing models with over 90% accuracy and a remarkably low false positive rate (<1.3%), providing a robust tool for artists to protect their work after publication.

Introduction

Powerful generative models like Stable Diffusion and Midjourney have revolutionized creative workflows but also enabled the unauthorized mimicry of individual artistic styles. Techniques such as LoRA and Dreambooth allow users to fine-tune models on small datasets, producing near-perfect imitations. Current defenses, including adversarial perturbations (e.g., Glaze, Mist) and watermarking, are proactive—they must be applied before an image is shared. These methods are often fragile against common image processing and cannot protect the vast amount of artwork already online.

Comparison of original artworks and their AI-generated mimics during training and fine-tuning processes.

To address these gaps, we propose a post-hoc auditing approach that allows artists to verify if a suspect model has already been trained on their work without requiring prior modification of their images.

The StyleAuditor Framework

StyleAuditor operates on the premise that an artist's style is a multi-layered signature that a model inevitably memorizes during fine-tuning. The framework consists of four primary components:

  1. Style Extraction: Utilizing a pre-trained VGG network, the system extracts multi-granularity style representations. Early layers capture low-level details (textures, brushstrokes), while deeper layers capture high-level abstractions (composition, shapes).
  2. Auxiliary Dataset Collection: The auditor is trained using original artwork and "mimicked" artwork generated by the suspect model. This process explicitly accounts for model distortion—the inherent noise or deviation between real paintings and AI imitations.
  3. Discriminator Training: A discriminator is trained to distinguish the target artist's style features from those of other artists. The loss function balances classification accuracy with a "distortion loss" to isolate stylistic theft from general AI artifacts.
  4. Black-Box Auditing: Using a CLIP-interrogator to generate prompts from original images, the artist queries the suspect model. If the outputs consistently align with the artist's unique style fingerprint, the model is flagged.
The StyleAuditor architecture, showing the flow from artwork to style extraction and the discriminator's role in identifying style representations.

Experimental Results

The framework was evaluated using Stable Diffusion v2.1 and a curated dataset of 30 artists. To ensure a rigorous test, we selected artwork confirmed to be absent from the model's original training set (CLIP-retrieval similarity < 0.8) using WikiArt and Artbench for negative samples.

Key performance metrics include:

  • Accuracy: Over 90% in detecting pirated styles.
  • False Positive Rate (FPR): Less than 1.3%, minimizing the risk of false accusations in legal contexts.
  • Discriminative Power: An Area Under the Curve (AUC) of 0.993, indicating exceptional separation between legitimate and pirated models.

Conclusion and Future Work

StyleAuditor represents a significant step toward intrinsic protection for artists. By focusing on the inherent stylistic features learned by models, it bypasses the limitations of proactive watermarking. As a work-in-progress, future research will investigate adversarial countermeasures, such as "diluting" fingerprints by mixing multiple styles, and optimizing the framework for large-scale production environments.

Summary updated

BibTeX
@misc{du2024wip,
  author = {Linkang Du and Zheng Zhu and Min Chen and Shouling Ji and Peng Cheng and Jiming Chen and Zhikun Zhang},
  title = {{WIP: Auditing Artist Style Pirate in Text-to-image Generation Models}},
  year = {2024},
  doi = {10.14722/aiscc.2024.23009},
}
Figure