Abstract: As diffusion models have achieved success in image generation tasks, many studies have extended them to other related fields like image editing. Unlike image generation, image editing aims to modify an image based on user requests while keeping other parts of the image unchanged. Among these, text-based image editing is the most representative task. Some studies have shown that diffusion models are vulnerable to backdoor attacks, where attackers may poison the training data to inject the backdoor into models. However, previous backdoor attacks on diffusion models primarily focus on image generation models without considering image editing models. Given that image editing models accept multimodal inputs, it raises a new question regarding the effectiveness of different modalities triggers in backdoor attacks on these models. To address this question, we propose a backdoor attack framework for image editing models, named TrojanEdit, which can handle different modalities triggers. We explore five types of visual triggers, three types of textual triggers, and combine them together as fifteen types of multimodal triggers, conducting extensive experiments for three types of backdoor attack goals. Our experimental results show that the image editing model has a backdoor bias for texture triggers. Compared to visual triggers, textual triggers have stronger attack effectiveness but also cause more damage to the model's normal functionality. Furthermore, we found that multimodal triggers can achieve a good balance between the attack effectiveness and model's normal functionality.
Introduction
Diffusion models have revolutionized image generation, and their success has naturally extended to image editing. Unlike standard generation, text-based image editing requires the model to modify specific elements of an image according to a prompt while preserving the rest of the context. However, this multimodal dependency—relying on both an image and a text instruction—introduces new security vulnerabilities.
While backdoor attacks have been explored in unimodal diffusion models (e.g., text-to-image), image editing models present a unique challenge. Attackers can poison training data to inject a backdoor that triggers malicious behavior only when specific "triggers" are present. TrojanEdit is the first framework designed to address these multimodal complexities, ensuring that a backdoor is only activated when both a visual patch and a textual prompt are provided.
The Challenge: Backdoor Modality Bias
A significant hurdle in backdooring multimodal models is modality bias. In a naive attack where both a text trigger (like a specific character) and a visual trigger (like a small square) are used, the model's optimization process often favors the "stronger" signal. Typically, the text modality dominates, leading the model to ignore the visual trigger or trigger the attack even when the visual component is missing. This "lazy" learning behavior not only makes the attack less precise but also degrades the model's performance on legitimate, clean editing tasks.
Methodology: TrojanEdit and BKG
To overcome modality bias, the authors propose TrojanEdit, which implements a "multimodal AND-gate" backdoor. The core innovation is the Backdoor Gradients (BKG) balancing technique. This method ensures that the model learns a balanced representation where both triggers are strictly necessary for activation.
BKG Multimodal Balance Learning
The BKG mechanism operates during the poisoning phase through three main steps:
- Gradient Decoupling: The system calculates separate gradients for the textual and visual triggers.
- Dynamic Rescaling: It determines a "discrepancy ratio" ($\rho$) between the gradient magnitudes of the two modalities.
- Rebalancing: By rescaling the updates, BKG prevents the stronger modality from suppressing the weaker one, forcing the model to rely on both inputs.
Experimental Evaluation
The researchers evaluated TrojanEdit using a subset of the LAION-400M dataset with a poisoning rate of 4%. The framework was tested across four state-of-the-art models: InstructPix2Pix, SDEdit-OC, T2L, and SDEdit-E.
The results demonstrated exceptional performance:
- High Effectiveness: TrojanEdit achieved an average Attack Success Rate (ASR) of 97.75%.
- Stealthiness: The Error Attack Rate (EAR), which measures accidental triggers on clean data, was only 0.25%, indicating high precision.
- Robustness: The attack remained highly effective even under common defensive transformations like image compression (60% quality) and model fine-pruning. Notably, it bypassed TIJO, a leading multimodal backdoor detection method.
Conclusion
TrojanEdit highlights a critical vulnerability in multimodal image editing models. By introducing BKG balance learning, the authors demonstrated that backdoors can be made both highly effective and remarkably stealthy. This work underscores the urgent need for new defense mechanisms that can detect and mitigate multimodal threats in the rapidly evolving landscape of diffusion-based AI.
Summary updated