Abstract
As Text-to-Image (T2I) models become increasingly sophisticated, the risk of generating harmful content—ranging from explicit pornography to political propaganda—grows. Existing safety benchmarks often suffer from narrow scopes, coarse annotations, and low prompt effectiveness. We introduce T2I-RiskyPrompt, a comprehensive benchmark featuring 6,432 high-quality risky prompts across a hierarchical taxonomy of 14 fine-grained categories. Our work also proposes a reason-driven detection method using Multimodal Large Language Models (MLLMs) to identify unsafe images with high accuracy.
Introduction: Addressing the Gaps in T2I Safety
The rapid adoption of models like Stable Diffusion and Midjourney has highlighted significant safety vulnerabilities. Current research into mitigating these risks faces three primary hurdles:
- Narrow Scope: Most datasets focus on "Not Safe For Work" (NSFW) content, neglecting copyright infringement or political sensitivities.
- Coarse Annotations: Labels are typically binary (safe/unsafe), lacking the context of why an image is problematic.
- Low Effectiveness: Many prompts in existing benchmarks fail to actually trigger the generation of risky content due to linguistic ambiguity.
The T2I-RiskyPrompt Framework
To address these issues, we developed a rigorous six-stage pipeline to construct the T2I-RiskyPrompt dataset.
Hierarchical Risk Taxonomy
We defined 6 primary categories and 14 subcategories, including "Political Metaphor," "Illegal Trade," and "Terrifying Content." This fine-grained approach allows for a more nuanced evaluation of model safety across diverse domains.
Dataset Construction and Reason-Driven Detection
The pipeline involves prompt polishing using a fine-tuned LLaMA-3 to ensure semantic potency and validity filtering through actual image generation. A key technical contribution is our reason-driven detection method. Instead of a simple binary classification, we provide MLLMs with detailed human-annotated rationales. This alignment helps the model focus on specific visual elements—such as a specific logo or a distorted facial expression—that constitute a safety violation.
Key Insights from the Benchmark
Our evaluation of eight T2I models and various defense/attack strategies yielded several critical insights:
- The Capability-Risk Paradox: Models with superior instruction-following capabilities (e.g., SD3) are paradoxically more susceptible to generating risky content because they interpret complex, harmful prompts more accurately.
- Defense Bias: Current safety measures are heavily optimized for pornographic content, leaving categories like "illegal trade" (e.g., drug paraphernalia) or copyright theft significantly more vulnerable.
- Filter Vulnerabilities: Keyword-based filters are easily bypassed by "pseudoword" attacks, while feature-based filters struggle against sophisticated metaphorical prompts.
- Safety vs. Quality: There is a measurable "performance tax"; fine-tuning for safety often results in a degradation of overall image generation quality, as reflected in higher FID scores.
Results and Performance
T2I-RiskyPrompt demonstrates superior quality compared to existing benchmarks:
- Effectiveness: It achieved a score of 0.741 (the proportion of prompts triggering risky images), nearly doubling the effectiveness of earlier benchmarks like I2P (0.320).
- Clarity: The dataset's Perplexity (PPL) is just 86, indicating high semantic clarity compared to scores over 2,500 in older datasets.
- Detection Accuracy: Our reason-driven MLLM approach achieved 91.8% accuracy in identifying risky images using a 3B parameter model, outperforming much larger models that lacked reason-driven instructions.
Conclusion and Future Directions
T2I-RiskyPrompt provides a robust foundation for evaluating and improving the safety of generative models. Future work will involve extending this benchmark to Text-to-Video (T2V) models and developing more precise "concept erasure" techniques that can mitigate specific risks without compromising the model's general utility.
Summary updated