Abstract
Text embedding serves not only as a core component of modern NLP models but also plays a pivotal role in multimodal systems such as text-to-image (T2I) models, significantly facilitating user-friendly image generation through natural language instructions. However, while offering convenience, it also introduces additional risks. Misalignment issues in T2I models, whether caused by unintentional user inputs or targeted attacks, can negatively impact the reliability and ethics of these models. In this paper, we introduce TEOI, which fully considers the continuity and distribution characteristics of text embeddings. The framework directly optimizes the embeddings using gradient-based methods and then inverts them to obtain misaligned prompts of discrete tokens. The TEOI framework supports both text-modal and multimodal misalignment attacks, revealing the vulnerabilities of multimodal models that rely on text embeddings. Our work highlights the potential risks associated with embedding-based text representations in prevailing T2I models and provides a foundation for further research into robust and secure text-to-image generation systems.
Introduction
As text-to-image (T2I) models like Stable Diffusion and DALL-E become mainstream, ensuring they do not generate harmful, biased, or "Not-Safe-For-Work" (NSFW) content is a top priority. Most current defenses rely on prompt filtering—essentially a "banned words" list that blocks inputs containing offensive terms. However, these models do not process raw text directly; they process text embeddings (high-dimensional vectors).
This paper addresses the "misalignment" problem: the possibility that a benign-looking text prompt can be mapped to a "harmful" embedding vector. If an attacker can find these "wolf in sheep's clothing" prompts, they can bypass filters entirely, rendering current safety mechanisms obsolete. We introduce TEOI, a framework that exposes this vulnerability by bridging the gap between what a prompt says and what the model "sees."
Methods
The TEOI framework utilizes a two-stage process to discover misaligned prompts:
- Text Embedding Optimization: Instead of heuristic searching, TEOI uses gradient-based optimization. Starting from a target image (e.g., an NSFW image or a specific face), the framework "nudges" the text embedding vector until the T2I model generates a visually similar output. To maintain readability, the authors use hybrid optimization, freezing a "semantic cue" and only optimizing a small portion of the prompt.
- Embedding Inversion: To convert these optimized vectors back into discrete, readable English tokens, TEOI employs a sophisticated inversion method based on the Vec2Text architecture.
A key innovation is the discovery that real text embeddings are sparse. By adding a distribution constraint loss (L1 norm) during optimization, TEOI ensures adversarial embeddings stay within the "natural" bounds of the model's latent space, making the attack significantly more effective.
Experimental Evaluation
The researchers evaluated TEOI across several standard and safety-focused benchmarks, including COCO for general tasks, the MMA (Multi-modal Adversarial) Dataset for NSFW content, and FFHQ for targeted identity manipulation. They also tested against extended NSFW categories using subsets of LAION and the Unsafe Diffusion dataset.
Key Results and Insights
TEOI achieved high success rates, with an average Attack Success Rate (ASR) of 77.2% on the COCO dataset and 66.4% on NSFW content, significantly outperforming previous methods like MMA-Diffusion.
The framework also proved powerful in multimodal attacks, where input image pixels and text prompts are optimized simultaneously. Even "Safe Stable Diffusion," a model fine-tuned for robustness, remained vulnerable to TEOI-generated content. These results highlight that current safety filters are often too shallow, focusing on keywords while ignoring the "bad intent" hidden in the mathematical representation of the prompt.
Conclusion
The success of TEOI demonstrates that the alignment between human language and model embeddings is fragile. The study suggests that future defenses should move away from simple keyword filtering and toward semantic filtering, which checks if the meaning of a prompt (its embedding) is too close to known harmful concepts. While currently a white-box attack, TEOI provides a foundation for further research into robust and secure text-to-image generation systems.
Summary updated