Key-Value Mapping-Based Text-to-Image Diffusion Model Backdoor Attacks

Lujia Chai, Y. Thomas Hou, Guozhao Liao, Qiuling YueAlgorithms 2026 · DOI

TL;DR

This paper introduces two novel backdoor attack methods, AttnBackdoor and SemBackdoor, that exploit the key-value mapping mechanism in text-to-image diffusion models. These attacks achieve high success rates with minimal parameter modifications, demonstrating significant security vulnerabilities in these powerful generative AI systems.

What Problem Does This Paper Address?

Text-to-image (T2I) diffusion models, like Stable Diffusion, have revolutionized content creation, but their rapid advancement has outpaced security research. A critical vulnerability lies within the Transformer architecture's key-value (KV) mapping, crucial for aligning text prompts with image generation. Existing backdoor attacks are often inefficient, require extensive data poisoning or fine-tuning, and lack stealth. This paper addresses the need for more efficient, stealthy, and precise backdoor attacks that target the core mechanisms of T2I models, thereby highlighting security blind spots and paving the way for better defenses.

Methods

The authors propose two novel backdoor attack methods, AttnBackdoor and SemBackdoor, both leveraging the Transformer's key-value mapping principle:

  • AttnBackdoor: This method targets the U-Net's cross-attention layers, specifically fine-tuning the key-value projection matrices. By modifying a small percentage (≈5%) of parameters, it injects instance-level backdoor mappings, allowing for precise control over generated images based on specific trigger prompts.
  • SemBackdoor: This approach focuses on the text encoder's MLP projection matrix, editing it to establish semantic-level mappings. With an even smaller parameter modification (≈0.3%), it achieves class-level backdoor triggers, influencing the semantic interpretation of prompts.

Both methods aim for efficiency and stealth by minimizing parameter updates and training time compared to prior work.

Datasets

  • Training/Attack Instance Data: The DreamBooth personalized dataset was used to select target concepts and construct corresponding text prompts for attack instance data.
  • Clean Data: For AttnBackdoor, clean samples were generated online by the original model to act as a constraint. For SemBackdoor, specific prompt texts were designed for semantic alignment.
  • Evaluation Data: The Pokemon-BLIP-Captions dataset was used for robustness testing. The MS-COCO 2014 validation set was used for FID score evaluation.

Key Insights and Ideas

  • Key-Value Mapping as a Unified Vulnerability: The paper establishes that the key-value storage and mapping mechanism within Transformer architectures is a fundamental security weakness in T2I diffusion models, exploitable for efficient backdoor injection.
  • Dual Attack Paradigms: AttnBackdoor and SemBackdoor represent complementary attack strategies. AttnBackdoor excels at instance-level precision through visual projection, while SemBackdoor achieves class-level attacks and better concealment via semantic alignment.
  • Minimal Perturbation, Maximum Impact: The research demonstrates that modifying a tiny fraction of parameters (as low as 0.3%) can lead to highly effective backdoor attacks, challenging traditional detection methods that rely on detecting large-scale weight anomalies.
  • Constraint Strategies Enhance Selectivity: Using clean data as a constraint in AttnBackdoor significantly improves trigger selectivity, preventing accidental activation and enhancing concealment.

Key Results

  • High Attack Success Rates: Both methods achieved impressive attack success rates (ASR), with SemBackdoor reaching 98.6% and AttnBackdoor 97.2%.
  • Efficiency Gains: Parameter updates and training times were reduced by 1-2 orders of magnitude compared to existing methods. SemBackdoor required only 133 seconds and 2.3 x 10^6 parameters.
  • Preservation of Benign Quality: The attacks maintained high generation quality and semantic consistency for benign prompts, making them difficult to detect. LPIPS scores indicated good concealment, with SemBackdoor achieving a better LPIPS value (0.11) than AttnBackdoor (0.25).
  • Robustness: AttnBackdoor showed stronger robustness against fine-tuning iterations compared to SemBackdoor, suggesting that backdoors integrated deeper into the generation path are more resilient.
  • Adaptability: Both methods demonstrated good adaptability across various trigger types, including semantically transparent and concealed triggers.

Limitations and Future Work

  • Model Architecture Generalizability: The experiments were primarily conducted on Stable Diffusion v1.4 and v2.1. Further evaluation is needed for larger models like DiT and SDXL, which may require adapting attack points and strategies.
  • Black-Box Scenarios: The current research focuses on white-box attacks. Investigating the feasibility of attacks in more constrained gray-box or black-box settings is crucial.
  • Defense Mechanisms: While the paper highlights vulnerabilities, it suggests that current detection strategies may be ineffective. Future work should focus on developing specialized detection and mitigation mechanisms, potentially by modeling statistical differences in sensitive parameter subspaces or analyzing attention maps.
  • Robustness of Backdoors: The paper notes that backdoors based on generation paths (like AttnBackdoor) might have stronger survivability than those based on semantic editing (like SemBackdoor) during downstream fine-tuning, posing challenges for defense.

Summary updated

Abstract

Text-to-image (T2I) generation, a core component of generative artificial intelligence(AI), is increasingly important for creative industries and human–computer interaction. Despite impressive progress in realism and diversity, diffusion models still exhibit critical security blind spots particularly in the Transformer key-value mapping mechanism that underpins cross-modal alignment. Existing backdoor attacks often rely on large-scale data poisoning or extensive fine-tuning, leading to low efficiency and limited stealth. To address these challenges, we propose two efficient backdoor attack methods AttnBackdoor and SemBackdoor grounded in the Transformer’s key-value storage principle. AttnBackdoor injects precise mappings between trigger prompts and target instances by fine-tuning the key-value projection matrices in U-Net cross-attention layers (≈5% of parameters). SemBackdoor establishes semantic-level mappings by editing the text encoder’s MLP projection matrix (≈0.3% of parameters). Both approaches achieve high attack success rates (>90%), with SemBackdoor reaching 98.6% and AttnBackdoor 97.2%. They also reduce parameter updates and training time by 1–2 orders of magnitude compared to prior work while preserving benign generation quality. Our findings reveal dual vulnerabilities at visual and semantic levels and provide a foundation for developing next generation defenses for secure generative AI.
BibTeX
@inproceedings{chai2026key,
  author = {Lujia Chai and Y. Thomas Hou and Guozhao Liao and Qiuling Yue},
  title = {{Key-Value Mapping-Based Text-to-Image Diffusion Model Backdoor Attacks}},
  booktitle = {Algorithms},
  year = {2026},
  doi = {10.3390/a19010074},
}
Figure