Abstract
Text-to-image (T2I) generation models like DALL-E 3 and Gemini have revolutionized content creation, but their safety remains a critical concern. While developers employ rigorous safeguards to prevent the generation of harmful content, these filters often rely on single-turn prompt evaluation. We introduce the Chain-of-Jailbreak (CoJ) attack, a novel method that bypasses safety filters by decomposing a single malicious query into a sequence of seemingly benign editing steps. Our evaluation on state-of-the-art models (GPT-4V, GPT-4o, Gemini 1.5, and Gemini 1.5 Pro) reveals that CoJ achieves a success rate of over 60%, far exceeding traditional jailbreaking methods. To mitigate this threat, we propose Think Twice Prompting, a defense mechanism that forces models to simulate the final image's semantic meaning before rendering, successfully blocking over 95% of CoJ attempts.
Introduction
As image generation models become integrated into professional workflows, ensuring they do not produce toxic, violent, or illegal content is paramount. Current safety architectures are largely "myopic," focusing on the immediate input prompt. This paper identifies a significant vulnerability in this approach: the lack of multi-turn context. By iteratively modifying an image through a series of prompts, an attacker can lead a model toward a prohibited output that would have been rejected if requested in a single step.
The Chain-of-Jailbreak (CoJ) Attack
The core of the CoJ attack is Query Decomposition. Instead of submitting a "one-shot" malicious prompt, the attacker breaks the request into sub-queries that appear harmless in isolation.
Inspired by the Levenshtein Distance, we propose three primary editing operations:
- Delete-then-Insert: Generate a "safe" version of a slogan by omitting a sensitive word, then ask the model to insert that word in a subsequent turn.
- Insert-then-Delete: This highly effective strategy involves adding a benign word (e.g., "not") to neutralize a harmful prompt (e.g., "Smoking is not good"), establishing a safe baseline. The attacker then prompts the model to delete the neutralizing word.
- Change-then-Change-Back: Swap a malicious term for a safe synonym, then "edit" it back to the original toxic term once the image state is established.
These operations can be applied at the word, character, or image level, allowing attackers to manipulate text slogans or visual elements (e.g., changing a harmless object into a weapon) step-by-step.
Experimental Setup and CoJ-Bench
To evaluate this vulnerability, we developed CoJ-Bench, a comprehensive dataset covering nine safety scenarios: Abusive, Pornography, Unlawful, Hate Speech, Bias, Physical Harm, Violence, Child Abuse, and Animal Abuse. We utilized an LLM (Mistral-Large-2) to automatically decompose 150 "seed" malicious queries—which were initially refused by all target models—into 1,070 multi-step editing sequences.
Results and Analysis
Our experiments demonstrate that CoJ is significantly more effective than traditional jailbreaking techniques like character roleplay or affirmation prefixes, which only achieved a ~14% success rate. In contrast, CoJ bypassed safeguards in over 60% of cases across the tested models.
Key findings include:
- Model Vulnerability: GPT-4o was the most susceptible, with a human-evaluated jailbreak success rate of 62.3%.
- Effectiveness of Steps: Success rates generally increase with the number of editing steps (peaking between 3-5 steps), as more granular changes are less likely to trigger "turn-based" filters.
- Word-Level Success: Editing at the word level proved more successful than character-level edits, as the latter often resembled the banned terms too closely, triggering basic keyword filters.
Defense: Think Twice Prompting
To counter the "step-by-step" trap, we propose Think Twice Prompting. This defense requires the model to internally simulate and describe the final image's content in text before the rendering process begins. This "mental rehearsal" forces the model to evaluate the cumulative semantic meaning of the conversation rather than just the delta of the latest prompt. In our tests, this method raised the defense success rate to an average of 97% across all models.
Conclusion
The Chain-of-Jailbreak attack highlights a critical flaw in current state-of-the-art image generation safeguards. By exploiting the lack of multi-turn contextual awareness, attackers can systematically generate harmful content. While our proposed Think Twice Prompting offers a robust defense, this work underscores the need for safety evaluations to evolve alongside the increasingly agentic and conversational nature of AI models.
Summary updated