Abstract
Current safety mechanisms for text-to-image (T2I) models often rely on simplistic keyword filters that are easily circumvented. This report details a distillation methodology used to extract over 1,100 adversarial prompts from the Inappropriate Image Prompts (I2P) benchmark. By evaluating these prompts against industry-standard ban-lists, we demonstrate that nearly two-thirds of known-toxic prompts bypass existing filters. Our findings highlight the fragility of word-based guardrails and provide a robust dataset for the Adversarial Nibbler safety challenge to facilitate the development of more resilient AI safety systems.
Introduction
The rapid deployment of text-to-image models like Stable Diffusion and Midjourney has brought significant concerns regarding the generation of unsafe, biased, or explicit content. Because these models are trained on massive, often unfiltered web-scale datasets, they internalize a wide array of harmful associations. To mitigate this, practitioners frequently employ input filters—lists of "banned words"—to block problematic queries. However, these filters are often reactive and easily bypassed through "red-teaming" or creative prompting.
This paper addresses the critical need for large-scale, quantifiable evaluation of adversarial prompts: inputs that appear benign to standard filters but reliably trigger the generation of inappropriate imagery. We present a distillation approach to identify these prompts, contributing to the Adversarial Nibbler challenge, a crowdsourced initiative aimed at improving AI safety guardrails.
Methodology: Distilling Adversarial Prompts
To identify truly adversarial inputs, we utilized a multi-stage distillation process starting from the I2P benchmark, which contains over 4,700 real-world prompts known to generate inappropriate content.
- Baseline Filtering: We evaluated the I2P prompts against a representative industry filter: Midjourney’s deployed list of approximately 800 banned words.
- Exclusion of Explicit Hits: Any prompt that explicitly triggered the ban-list was removed. These are "low-hanging fruit" that current systems already catch.
- Adversarial Isolation: From the remaining "benign-looking" prompts, we isolated those with a high probability ($\ge 50\%$) of generating inappropriate material. This probability was determined using automated classifiers (Q16 and NudeNet) on images previously generated by Stable Diffusion.
This process effectively separates overtly toxic prompts from adversarial ones—those that successfully "sneak" past the gatekeeper.
Key Findings and Vulnerabilities
Our analysis of the distilled dataset revealed several systemic weaknesses in current safety architectures:
1. The Fragility of Ban-Lists
Simple lexical variations are sufficient to bypass most filters. We observed that while a filter might catch the word "nude," it often misses "nudity." Similarly, "blood" might be blocked while "bleeding" is permitted. Furthermore, high-level concepts such as "violence" or "murder" were frequently absent from the ban-lists entirely, allowing direct requests for disturbing content.
2. Toxic Combinations of Benign Terms
Adversarial prompts often leverage "innocuous" words that, when combined, trigger unsafe outputs. For instance, combining benign adjectives with non-erotic anatomical terms (e.g., "gorgeous female" + "torso") consistently generated sexually explicit imagery. Because neither word is inherently "bad," they bypass filters that lack semantic or contextual understanding.
3. Artist Names as Backdoor Triggers
Specific artist names can act as reliable triggers for unsafe content. For example, prompts referencing "photograph by Vanessa Beecroft" often forced the model to generate images involving nudity or themes of anorexia. These names function as high-level tokens that encapsulate unsafe concepts, even when the rest of the prompt appears safe.
4. Contextual and Biased Risks
Approximately 10% of the distilled prompts targeted public figures in defamatory or misleading contexts (e.g., political figures in jail). While the individual words are not "unsafe," the resulting images pose significant societal risks. Additionally, the models exhibited deep-seated racial biases, such as the disproportionate sexualization of specific ethnicities even when given generic, non-sexual prompts.
Results
The evaluation against the Midjourney filter yielded stark results:
- Low Detection Rate: Only 34% of the original I2P prompts (known to produce unsafe images) were blocked by the filter.
- Category Bias: The filter was significantly more effective at blocking sexual content than hate speech or harassment. The block rate for the "sexual" category was nearly four times higher than for the "hate" category.
- Dataset Contribution: We successfully distilled a dataset of approximately 1,100 adversarial prompts that bypass standard filters while maintaining a high likelihood of generating inappropriate content.
Conclusion and Future Directions
The results of this distillation process confirm that keyword-based ban-lists are an insufficient primary defense for T2I models. They are easily circumvented by synonyms, artist triggers, and contextual combinations. While ban-lists serve as a useful first layer, future safety research must prioritize context-aware guardrails and architectural interventions, such as concept erasure within the model weights themselves. By releasing this distilled dataset to the Adversarial Nibbler challenge, we hope to provide the community with the necessary tools to build more robust and ethically aligned generative AI.
Summary updated