T2I-RiskyPrompt: A Benchmark for Safety Evaluation, Attack, and Defense on Text-to-Image Model

Chenyu Zhang, Tairen Zhang, Lanjun Wang, Ruidong Chen, Wenhui Li, Anan Liu

Abstract

As Text-to-Image (T2I) models become increasingly sophisticated, the risk of generating harmful content—ranging from explicit pornography to political propaganda—grows. Existing safety benchmarks often suffer from narrow scopes, coarse annotations, and low prompt effectiveness. We introduce T2I-RiskyPrompt, a comprehensive benchmark featuring 6,432 high-quality risky prompts across a hierarchical taxonomy of 14 fine-grained categories. Our work also proposes a reason-driven detection method using Multimodal Large Language Models (MLLMs) to identify unsafe images with high accuracy.

Introduction: Addressing the Gaps in T2I Safety

The rapid adoption of models like Stable Diffusion and Midjourney has highlighted significant safety vulnerabilities. Current research into mitigating these risks faces three primary hurdles:

  1. Narrow Scope: Most datasets focus on "Not Safe For Work" (NSFW) content, neglecting copyright infringement or political sensitivities.
  2. Coarse Annotations: Labels are typically binary (safe/unsafe), lacking the context of why an image is problematic.
  3. Low Effectiveness: Many prompts in existing benchmarks fail to actually trigger the generation of risky content due to linguistic ambiguity.

The T2I-RiskyPrompt Framework

To address these issues, we developed a rigorous six-stage pipeline to construct the T2I-RiskyPrompt dataset.

Hierarchical Risk Taxonomy

We defined 6 primary categories and 14 subcategories, including "Political Metaphor," "Illegal Trade," and "Terrifying Content." This fine-grained approach allows for a more nuanced evaluation of model safety across diverse domains.

Figure 1: The T2I-RiskyPrompt taxonomy and annotated examples including prompts, classes, and detailed risk reasons.

Dataset Construction and Reason-Driven Detection

The pipeline involves prompt polishing using a fine-tuned LLaMA-3 to ensure semantic potency and validity filtering through actual image generation. A key technical contribution is our reason-driven detection method. Instead of a simple binary classification, we provide MLLMs with detailed human-annotated rationales. This alignment helps the model focus on specific visual elements—such as a specific logo or a distorted facial expression—that constitute a safety violation.

Key Insights from the Benchmark

Our evaluation of eight T2I models and various defense/attack strategies yielded several critical insights:

  • The Capability-Risk Paradox: Models with superior instruction-following capabilities (e.g., SD3) are paradoxically more susceptible to generating risky content because they interpret complex, harmful prompts more accurately.
  • Defense Bias: Current safety measures are heavily optimized for pornographic content, leaving categories like "illegal trade" (e.g., drug paraphernalia) or copyright theft significantly more vulnerable.
  • Filter Vulnerabilities: Keyword-based filters are easily bypassed by "pseudoword" attacks, while feature-based filters struggle against sophisticated metaphorical prompts.
  • Safety vs. Quality: There is a measurable "performance tax"; fine-tuning for safety often results in a degradation of overall image generation quality, as reflected in higher FID scores.

Results and Performance

T2I-RiskyPrompt demonstrates superior quality compared to existing benchmarks:

  • Effectiveness: It achieved a score of 0.741 (the proportion of prompts triggering risky images), nearly doubling the effectiveness of earlier benchmarks like I2P (0.320).
  • Clarity: The dataset's Perplexity (PPL) is just 86, indicating high semantic clarity compared to scores over 2,500 in older datasets.
  • Detection Accuracy: Our reason-driven MLLM approach achieved 91.8% accuracy in identifying risky images using a 3B parameter model, outperforming much larger models that lacked reason-driven instructions.

Conclusion and Future Directions

T2I-RiskyPrompt provides a robust foundation for evaluating and improving the safety of generative models. Future work will involve extending this benchmark to Text-to-Video (T2V) models and developing more precise "concept erasure" techniques that can mitigate specific risks without compromising the model's general utility.

Summary updated

Abstract

Using risky text prompts, such as pornography and violent prompts, to test the safety of text-to-image (T2I) models is a critical task. However, existing risky prompt datasets are limited in three key areas: 1) limited risky categories, 2) coarse-grained annotation, and 3) low effectiveness. To address these limitations, we introduce T2I-RiskyPrompt, a comprehensive benchmark designed for evaluating safety-related tasks in T2I models. Specifically, we first develop a hierarchical risk taxonomy, which consists of 6 primary categories and 14 fine-grained subcategories. Building upon this taxonomy, we construct a pipeline to collect and annotate risky prompts. Finally, we obtain 6,432 effective risky prompts, where each prompt is annotated with both hierarchical category labels and detailed risk reasons. Moreover, to facilitate the evaluation, we propose a reason-driven risky image detection method that explicitly aligns the MLLM with safety annotations. Based on T2I-RiskyPrompt, we conduct a comprehensive evaluation of eight T2I models, nine defense methods, five safety filters, and five attack strategies, offering nine key insights into the strengths and limitations of T2I model safety. Finally, we discuss potential applications of T2I-RiskyPrompt across various research fields. The dataset and code are provided in https://github.com/datar001/T2I-RiskyPrompt.
BibTeX
@inproceedings{zhang2026t,
  author = {Chenyu Zhang and Tairen Zhang and Lanjun Wang and Ruidong Chen and Wenhui Li and Anan Liu},
  title = {{T2I-RiskyPrompt: A Benchmark for Safety Evaluation, Attack, and Defense on Text-to-Image Model}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2026},
  url = {https://openreview.net/forum?id=P3PqmGpjFi},
}
Figure