Chain-of-Jailbreak Attack for Image Generation Models via Editing Step by Step

Wenxuan Wang, Kuiyi Gao, Zihan Jia, Youliang Yuan, Jen-tse Huang, Qiuzhi Liu, Shuai Wang, Wenxiang Jiao, Zhaopeng Tu

Abstract

Text-to-image (T2I) generation models like DALL-E 3 and Gemini have revolutionized content creation, but their safety remains a critical concern. While developers employ rigorous safeguards to prevent the generation of harmful content, these filters often rely on single-turn prompt evaluation. We introduce the Chain-of-Jailbreak (CoJ) attack, a novel method that bypasses safety filters by decomposing a single malicious query into a sequence of seemingly benign editing steps. Our evaluation on state-of-the-art models (GPT-4V, GPT-4o, Gemini 1.5, and Gemini 1.5 Pro) reveals that CoJ achieves a success rate of over 60%, far exceeding traditional jailbreaking methods. To mitigate this threat, we propose Think Twice Prompting, a defense mechanism that forces models to simulate the final image's semantic meaning before rendering, successfully blocking over 95% of CoJ attempts.

Introduction

As image generation models become integrated into professional workflows, ensuring they do not produce toxic, violent, or illegal content is paramount. Current safety architectures are largely "myopic," focusing on the immediate input prompt. This paper identifies a significant vulnerability in this approach: the lack of multi-turn context. By iteratively modifying an image through a series of prompts, an attacker can lead a model toward a prohibited output that would have been rejected if requested in a single step.

The Chain-of-Jailbreak (CoJ) Attack

The core of the CoJ attack is Query Decomposition. Instead of submitting a "one-shot" malicious prompt, the attacker breaks the request into sub-queries that appear harmless in isolation.

The Chain-of-Jailbreak (CoJ) attack decomposes a rejected malicious prompt into a sequence of iterative editing steps to bypass safety filters.

Inspired by the Levenshtein Distance, we propose three primary editing operations:

  1. Delete-then-Insert: Generate a "safe" version of a slogan by omitting a sensitive word, then ask the model to insert that word in a subsequent turn.
  2. Insert-then-Delete: This highly effective strategy involves adding a benign word (e.g., "not") to neutralize a harmful prompt (e.g., "Smoking is not good"), establishing a safe baseline. The attacker then prompts the model to delete the neutralizing word.
  3. Change-then-Change-Back: Swap a malicious term for a safe synonym, then "edit" it back to the original toxic term once the image state is established.

These operations can be applied at the word, character, or image level, allowing attackers to manipulate text slogans or visual elements (e.g., changing a harmless object into a weapon) step-by-step.

Experimental Setup and CoJ-Bench

To evaluate this vulnerability, we developed CoJ-Bench, a comprehensive dataset covering nine safety scenarios: Abusive, Pornography, Unlawful, Hate Speech, Bias, Physical Harm, Violence, Child Abuse, and Animal Abuse. We utilized an LLM (Mistral-Large-2) to automatically decompose 150 "seed" malicious queries—which were initially refused by all target models—into 1,070 multi-step editing sequences.

Results and Analysis

Our experiments demonstrate that CoJ is significantly more effective than traditional jailbreaking techniques like character roleplay or affirmation prefixes, which only achieved a ~14% success rate. In contrast, CoJ bypassed safeguards in over 60% of cases across the tested models.

Jailbreak success rates for the CoJ attack across nine safety categories, showing high vulnerability in GPT-4o and GPT-4V.

Key findings include:

  • Model Vulnerability: GPT-4o was the most susceptible, with a human-evaluated jailbreak success rate of 62.3%.
  • Effectiveness of Steps: Success rates generally increase with the number of editing steps (peaking between 3-5 steps), as more granular changes are less likely to trigger "turn-based" filters.
  • Word-Level Success: Editing at the word level proved more successful than character-level edits, as the latter often resembled the banned terms too closely, triggering basic keyword filters.

Defense: Think Twice Prompting

To counter the "step-by-step" trap, we propose Think Twice Prompting. This defense requires the model to internally simulate and describe the final image's content in text before the rendering process begins. This "mental rehearsal" forces the model to evaluate the cumulative semantic meaning of the conversation rather than just the delta of the latest prompt. In our tests, this method raised the defense success rate to an average of 97% across all models.

Conclusion

The Chain-of-Jailbreak attack highlights a critical flaw in current state-of-the-art image generation safeguards. By exploiting the lack of multi-turn contextual awareness, attackers can systematically generate harmful content. While our proposed Think Twice Prompting offers a robust defense, this work underscores the need for safety evaluations to evolve alongside the increasingly agentic and conversational nature of AI models.

Summary updated

Abstract

Text-based image generation models, such as Stable Diffusion and DALL-E 3, hold significant potential in content creation and publishing workflows, making them the focus in recent years. Despite their remarkable capability to generate diverse and vivid images, considerable efforts are being made to prevent the generation of harmful content, such as abusive, violent, or pornographic material. To assess the safety of existing models, we introduce a novel jailbreaking method called Chain-of-Jailbreak (CoJ) attack, which compromises image generation models through a step-by-step editing process. Specifically, for malicious queries that cannot bypass the safeguards with a single prompt, we intentionally decompose the query into multiple sub-queries. The image generation models are then prompted to generate and iteratively edit images based on these sub-queries. To evaluate the effectiveness of our CoJ attack method, we constructed a comprehensive dataset, CoJ-Bench, encompassing nine safety scenarios, three types of editing operations, and three editing elements. Experiments on four widely-used image generation services provided by GPT-4V, GPT-4o, Gemini 1.5 and Gemini 1.5 Pro, demonstrate that our CoJ attack method can successfully bypass the safeguards of models for over 60% cases, which significantly outperforms other jailbreaking methods (i.e., 14%). Further, to enhance these models' safety against our CoJ attack method, we also propose an effective prompting-based method, Think Twice Prompting, that can successfully defend over 95% of CoJ attack. We release our dataset and code to facilitate the AI safety research.
BibTeX
@inproceedings{wang2025chain,
  author = {Wenxuan Wang and Kuiyi Gao and Zihan Jia and Youliang Yuan and Jen-tse Huang and Qiuzhi Liu and Shuai Wang and Wenxiang Jiao and Zhaopeng Tu},
  title = {{Chain-of-Jailbreak Attack for Image Generation Models via Editing Step by Step}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2025},
}
Figure