T2I-RiskyPrompt: A Benchmark for Safety Evaluation, Attack, and Defense on Text-to-Image Model

Chenyu Zhang, Tairen Zhang, Lanjun Wang, Ruidong Chen, Wenhui Li, An-An Liu

Abstract

Using risky text prompts, such as pornography and violent prompts, to test the safety of text-to-image (T2I) models is a critical task. However, existing risky prompt datasets are limited in three key areas: 1) limited risky categories, 2) coarse-grained annotation, and 3) low effectiveness. To address these limitations, we introduce T2I-RiskyPrompt, a comprehensive benchmark designed for evaluating safety-related tasks in T2I models. Specifically, we first develop a hierarchical risk taxonomy, which consists of 6 primary categories and 14 fine-grained subcategories. Building upon this taxonomy, we construct a pipeline to collect and annotate risky prompts. Finally, we obtain 6,432 effective risky prompts, where each prompt is annotated with both hierarchical category labels and detailed risk reasons. Moreover, to facilitate the evaluation, we propose a reason-driven risky image detection method that explicitly aligns the MLLM with safety annotations. Based on T2I-RiskyPrompt, we conduct a comprehensive evaluation of eight T2I models, nine defense methods, five safety filters, and five attack strategies, offering nine key insights into the strengths and limitations of T2I model safety.

Addressing the Safety Gap in T2I Models

As Text-to-Image (T2I) models like Stable Diffusion and Midjourney become mainstream, their potential to generate harmful content—ranging from explicit violence to political misinformation and copyright infringement—has become a major concern. Existing safety benchmarks often suffer from narrow scopes (focusing primarily on NSFW content), coarse binary annotations, and low effectiveness, where "risky" prompts fail to actually trigger harmful generations in modern models.

T2I-RiskyPrompt addresses these flaws by providing 6,432 high-effectiveness prompts across 14 fine-grained categories. Beyond simple labeling, the dataset includes detailed human-annotated rationales for why an image is unsafe, enabling a new "reason-driven" detection method.

Methodology and Taxonomy

The researchers developed a rigorous six-stage pipeline to build T2I-RiskyPrompt, involving collection from real-world databases, GPT-4o-assisted polishing, and diversity filtering. Crucially, they implemented validity filtering by generating images with state-of-the-art models (SD3 and FLUX) and discarding prompts that failed to produce intended risky visual elements.

Overview of the T2I-RiskyPrompt hierarchical risk taxonomy and annotated examples showing prompts, classes, and risk reasons.

The taxonomy includes six primary categories: Pornography, Violence, Disturbing Content, Illegal Activities, Copyright Infringement, and Political Sensitivity. To leverage these annotations, the authors proposed a reason-driven risky image detection method. Instead of a simple classifier, they prompt a Multimodal Large Language Model (MLLM) with the specific "risk rationale" to determine if a generated image violates safety policies.

Key Insights and Evaluation

Through comprehensive testing of 8 T2I models and various defense/attack strategies, several key insights emerged:

  • The Capability-Risk Paradox: Models with stronger generative capabilities (like SD3 and FLUX) are often more dangerous because they follow instructions more precisely, rendering nuanced risky content that older models might fail to visualize.
  • Pornography Bias: Developers have prioritized filtering pornography, often at the expense of other risks. For instance, the PixArt model shows low risk for porn but remains highly susceptible to violent or politically sensitive imagery.
  • Multi-Concept Struggles: Current defense methods, such as concept erasing, work well for single risks but struggle when a prompt combines multiple risky concepts simultaneously.
  • Linguistic "Camouflage": Jailbreaking attacks using metaphors or "pseudowords" easily bypass keyword-based filters but are often caught by MLLM-based vision filters that analyze the final visual output.

Results and Performance

The proposed reason-driven detection method achieved 91.8% accuracy using only a 3B-parameter MLLM, outperforming much larger standard detectors. T2I-RiskyPrompt also demonstrated superior benchmark quality with a Perplexity (PPL) of 86 and a high effectiveness score of 0.741, meaning its prompts are significantly more likely to successfully trigger risky generations compared to previous benchmarks.

Conclusion

T2I-RiskyPrompt provides a robust framework for evaluating and improving T2I safety. While manual annotation is resource-intensive, the detailed risk reasons offer a path forward for "unlearning" harmful concepts more precisely during model training and extending safety evaluations to emerging fields like Text-to-Video (T2V).

Summary updated

Abstract

Using risky text prompts, such as pornography and violent prompts, to test the safety of text-to-image (T2I) models is a critical task. However, existing risky prompt datasets are limited in three key areas: 1) limited risky categories, 2) coarse-grained annotation, and 3) low effectiveness. To address these limitations, we introduce T2I-RiskyPrompt, a comprehensive benchmark designed for evaluating safety-related tasks in T2I models. Specifically, we first develop a hierarchical risk taxonomy, which consists of 6 primary categories and 14 fine-grained subcategories. Building upon this taxonomy, we construct a pipeline to collect and annotate risky prompts. Finally, we obtain 6,432 effective risky prompts, where each prompt is annotated with both hierarchical category labels and detailed risk reasons. Moreover, to facilitate the evaluation, we propose a reason-driven risky image detection method that explicitly aligns the MLLM with safety annotations. Based on T2I-RiskyPrompt, we conduct a comprehensive evaluation of eight T2I models, nine defense methods, five safety filters, and five attack strategies, offering nine key insights into the strengths and limitations of T2I model safety. Finally, we discuss potential applications of T2I-RiskyPrompt across various research fields.
BibTeX
@inproceedings{zhang2026t,
  author = {Chenyu Zhang and Tairen Zhang and Lanjun Wang and Ruidong Chen and Wenhui Li and An-An Liu},
  title = {{T2I-RiskyPrompt: A Benchmark for Safety Evaluation, Attack, and Defense on Text-to-Image Model}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2026},
  url = {https://openreview.net/forum?id=P3PqmGpjFi},
}
Figure