Not with my name! Inferring artists' names of input strings employed by Diffusion Models

Roberto Leotta, Oliver Giudice, Luca Guarnera, Sebastiano BattiatoarXiv 2023 · arXiv

This paper, titled "Not with my name! Inferring artists’ names of input strings employed by Diffusion Models," tackles a burgeoning legal and ethical crisis in the AI era: the unauthorized use of an artist's style and name in generative prompts. As Diffusion Models (DMs) like DALL-E 2 become ubiquitous, the authors propose a forensic method to prove whether a specific artist's name was used to generate a given image, providing a potential tool for intellectual property (IP) protection.

TL;DR

Researchers have developed a forensic approach using Siamese Neural Networks to detect if a specific artist's name was included in the text prompt used to generate an AI image. By learning a dedicated similarity metric between real artworks and synthetic outputs, the system can estimate the probability of copyright-infringing "style mimicry."

What Problem Does This Paper Address?

The core conflict lies in how Diffusion Models are trained. To generate an image "in the style of Van Gogh," the model must have been trained on Van Gogh’s actual works, often without explicit consent. When a user prompts a model with an artist's name, the model draws upon these learned features to replicate their aesthetic.

From a legal standpoint, proving that a model was prompted with a specific name is difficult once the image is generated. Current tools can often distinguish "AI vs. Real," but they struggle to attribute which specific artist's identity was exploited in the prompt. This paper aims to bridge that gap, moving from general detection to specific forensic attribution.

Methods

The authors propose a multi-stage approach centered on Metric Learning:

  1. Baseline Classification: They first trained a standard ResNet-18 to classify original artworks by five famous artists. Interestingly, they found that a model trained only on real art failed to accurately classify synthetic art (a failure of transfer learning), suggesting that DMs introduce unique, invisible artifacts.
  2. Mixed Training: To fix this, they trained a new ResNet-18 on a mixed dataset of both real and synthetic images. This allowed the model to learn both the artistic style and the "fingerprint" of the DALL-E 2 generation process.
  3. Siamese Neural Network (SNN): The final, most robust approach uses a Siamese architecture (see Figure 1). Instead of simple classification, the SNN learns a distance metric ($d$). It takes pairs of images and learns to minimize the distance between images belonging to the same artist (whether real or synthetic) and maximize the distance between different artists.

![Fig.

  1. The proposed approach, moving from a baseline classifier to a Siamese engine for better generalization.](/figures/art_040aa6e8007d47e191ea012b808afb39)

Datasets

The study focused on five renowned artists: Alfred Sisley, Claude Monet, Pablo Picasso, Paul Cezanne, and Pierre-Auguste Renoir.

  • Original Data: Paintings were sourced from WikiArt.
  • Synthetic Data: 2,000 images per artist were generated using DALL-E 2. Prompts followed a template: "[Subject description], by [Artist Name]".
  • Validation: The synthetic dataset was manually cleaned to remove "outliers" (e.g., photo-realistic images that didn't match the intended painterly style).

Key Insights and Ideas

  • The Transfer Learning Gap: A key finding was that features learned from real oil paintings do not perfectly map to AI-generated "paintings." As shown in Figure 2, the model trained only on original works (a) struggled significantly compared to the model trained on a mixed dataset (b). This implies DMs alter images in subtle, invisible ways (likely in the frequency domain) that standard classifiers miss.
  • Prompt Attribution vs. Style Recognition: The authors argue their metric doesn't just look at "style" (brushstrokes, color) but specifically detects the probability that the name was in the prompt. This is a crucial distinction for forensic investigators.
  • Entropy and Data Scarcity: The authors noted higher "entropy" (confusion) for artists with fewer publicly available works, suggesting that DALL-E 2’s ability to mimic an artist is directly tied to the cardinality of that artist's presence in its training set.

![Fig.

  1. Confusion matrices showing the failure of transfer learning (a) vs. the success of the mixed-dataset approach (b).](/figures/art_c09e60c209b2431ca623829e60904f2c)

Key Results

The Siamese approach proved highly effective at "retrieval" tasks. When presented with a query image, the system could successfully identify the correct artist among the top results.

  • Probability Estimation: The authors defined a likelihood function where a distance close to 0 indicates a high probability the artist's name was used.
  • Convergence: The Siamese network showed stable convergence during training, as evidenced by the loss curves in Figure 3, indicating the learned metric is robust.
  • Case Study: In one example, the system assigned a 99.99% probability that the name "Picasso" was used to generate a synthetic image of children playing on a beach, despite the image not being a direct copy of any existing Picasso work.

![Fig.

  1. Training and validation loss for the Siamese architecture, showing successful metric learning.](/figures/art_389d1e0ee9a54a1ba92a7ce43c037c1f)

Limitations and Future Work

While promising, the study is a "preliminary" effort with several constraints:

  • Scope: It only tested five artists and one generative model (DALL-E 2). The authors admit the results might not generalize to GANs or other DMs like Stable Diffusion without specific retraining.
  • Reference Dependency: The method requires a reference set of both real and synthetic images for every artist one wishes to "protect."
  • Prompt Complexity: The study used simple prompts. Future work needs to investigate if the method holds up when an artist's name is buried in a long, complex prompt or mixed with other styles (e.g., "in the style of Picasso and Dali").

Ultimately, this research provides a vital starting point for Multimedia Forensics in the age of generative AI, offering a path toward technical solutions for the complex legal questions surrounding AI and art.

Summary updated

Abstract

Diffusion Models (DM) are highly effective at generating realistic, high-quality images. However, these models lack creativity and merely compose outputs based on their training data, guided by a textual input provided at creation time. Is it acceptable to generate images reminiscent of an artist, employing his name as input? This imply that if the DM is able to replicate an artist's work then it was trained on some or all of his artworks thus violating copyright. In this paper, a preliminary study to infer the probability of use of an artist's name in the input string of a generated image is presented. To this aim we focused only on images generated by the famous DALL-E 2 and collected images (both original and generated) of five renowned artists. Finally, a dedicated Siamese Neural Network was employed to have a first kind of probability. Experimental results demonstrate that our approach is an optimal starting point and can be employed as a prior for predicting a complete input string of an investigated image. Dataset and code are available at: https://github.com/ictlab-unict/not-with-my-name .
BibTeX
@misc{leotta2023not,
  author = {Roberto Leotta and Oliver Giudice and Luca Guarnera and Sebastiano Battiato},
  title = {{Not with my name! Inferring artists' names of input strings employed by Diffusion Models}},
  year = {2023},
  eprint = {2307.13527},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2307.13527},
}
Figure