Adversarial Attacks on Image Generation With Made-Up Words

Raphaël MillièrearXiv 2022 · DOI · arXiv

Abstract

This paper investigates a novel class of adversarial attacks on text-to-image (T2I) models, such as DALL-E 2, using "macaronic" and "evocative" prompts. By exploiting subword tokenization and multilingual training data, these attacks bypass standard keyword-based content filters. We demonstrate that nonsensical strings can consistently trigger specific visual concepts, highlighting a systemic vulnerability in large-scale vision-language models where the tokenizer acts as an unintended backdoor for obfuscated content generation.

Introduction

As text-to-image (T2I) models become more sophisticated, the risk of generating harmful, offensive, or misleading content grows. To mitigate this, developers implement "prompt filters"—blacklists of forbidden words. However, recent research suggests that models like DALL-E 2 may possess a "hidden vocabulary" where seemingly gibberish strings trigger specific images. This paper provides a systematic methodology for crafting these adversarial "nonce" strings to understand how models associate linguistic sub-units with visual concepts and how these associations can be weaponized to circumvent safety protocols.

Methodology

The study introduces two primary techniques for generating adversarial prompts that remain "encrypted" from simple text filters:

Macaronic Prompting

This involves "lexical hybridization," creating a new word by stitching together pieces of words from different languages that share the same meaning. For example, to evoke "birds," a user might combine parts of Vögel (German), uccelli (Italian), oiseaux (French), and pájaros (Spanish) to create the string uccoisegeljaros. Because the model's Byte Pair Encoding (BPE) tokenizer breaks words into sub-word units, it recognizes the constituent "bird" tokens from various languages even if the full string is gibberish to a human.

Diverse bird species generated using the macaronic prompt 'uccoisegeljaros'.

Evocative Prompting

This technique relies on morphological similarity rather than direct translation. It uses "pseudo-words" that mimic specific categories, such as:

  • Pseudo-Latin: Mimicking biological nomenclature (e.g., ceralineus rabaventis) to trigger organism imagery.
  • Pseudo-Pharmaceuticals: Using suffixes like -axin or -ofen to trigger medical imagery.
  • Geographical Suffixes: Using endings like -büchel (Germanic) to evoke specific regional architectures.

Analysis of Vulnerabilities

The fundamental vulnerability lies in the tokenizer. Models trained on massive, uncurated multilingual datasets learn that sub-word tokens for the same concept across different languages map to the same visual space. Macaronic prompting "packs" these tokens into a single string that bypasses filters looking for specific English keywords. Furthermore, these prompts often exhibit cross-model vulnerability, working across different architectures like DALL-E 2 and DALL-E mini, suggesting a systemic feature of how large-scale vision-language models learn from internet-scale data.

Results and Discussion

Qualitative evaluations show high consistency in generating intended imagery. Macaronic prompts like insekafetti (bugs) and avflugzereo (planes) consistently produced the intended results across multiple runs. The models also demonstrated successful compositional prompting, where multiple nonce words were combined in a single scene (e.g., "An eidelucertlagarzard eating a maripofarterling"), proving that these hidden concepts can interact syntactically.

A high-fidelity European Goldfinch generated through targeted adversarial subword manipulation.

While many prompts transferred between models, some failures occurred. For instance, a prompt producing butterflies on DALL-E 2 might produce mushrooms on DALL-E mini. This suggests that larger models may actually be more vulnerable to these attacks because they have learned more robust, fine-grained multilingual associations.

Conclusion and Defense Strategies

Keyword blacklists are fundamentally insufficient for securing T2I models against adversarial nonce words. Future defense strategies must move beyond text-based filtering toward visual filtering (checking the output of the model), vocabulary constraints (limiting prompts to standard dictionaries), or recursive checking (using captioning models to verify that generated images do not contain blacklisted concepts).

Summary updated

Abstract

Text-guided image generation models can be prompted to generate images using nonce words adversarially designed to robustly evoke specific visual concepts. Two approaches for such generation are introduced: macaronic prompting, which involves designing cryptic hybrid words by concatenating subword units from different languages; and evocative prompting, which involves designing nonce words whose broad morphological features are similar enough to that of existing words to trigger robust visual associations. The two methods can also be combined to generate images associated with more specific visual concepts. The implications of these techniques for the circumvention of existing approaches to content moderation, and particularly the generation of offensive or harmful images, are discussed.
BibTeX
@misc{milliere2022adversarial,
  author = {Raphaël Millière},
  title = {{Adversarial Attacks on Image Generation With Made-Up Words}},
  year = {2022},
  eprint = {2208.04135},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2208.04135},
}
Figure