Chat models that refuse a harmful request typed as text will often comply when the same request arrives with an image attached. The request might be rendered as a picture of words, split so the image shows an object and the text says "how do I mix this with bleach," or paired with an image carrying invisible pixel noise. Current defenses treat each of these attacks separately. They also tend to be expensive, because they re-run a gradient-based attacker inside the training loop, and they tend to be narrow, because they generalize poorly beyond the attack they were trained against. We propose one cheap fine-tuning objective aimed at all of them. Its starting point is that the model already knows how to refuse the text version of the request. Training should therefore make the model's behavior on the image-plus-text version match its own behavior on the text-only version, even when an adversary is allowed to push the image's internal representation as far as a budget allows.
The key design choice is where that adversary acts. A vision-language model (VLM) turns an image into a short sequence of "visual tokens," which are vectors produced by an image encoder and a small projection layer and then fed to the language model alongside the text. Every image-borne attack must pass through these vectors to reach the language model: typographic prompts, cross-modal compositions, and pixel noise alike. We therefore run the inner attacker directly on the visual tokens. This skips back-propagation through the vision encoder, needs no attack templates, and replaces preference labels with a free supervisory signal, namely the model's own text-only response. The bet is that robustness learned against a worst-case perturbation at this common chokepoint transfers to attack families never seen in training. We test that bet against attacks the training data deliberately excludes.
Core Claim
Several attacks fool VLMs: typographic jailbreaks such as FigStep, composite scenes as in MM-SafetyBench, the safe-image/safe-text compositions catalogued by USB, and bounded pixel attacks of the kind Schlarmann and Hein used on OpenFlamingo. Our hypothesis is that these attacks share a mechanism. Each produces visual tokens that suppress a refusal behavior the language backbone still expresses on the equivalent text. Images are Achilles' Heel of Alignment supports this reading, since it finds that even a blank image weakens text guardrails. If the hypothesis holds, one objective should close much of the gap, and it should do so across attack families rather than within one. That objective ties image-plus-text behavior to the model's own text-only behavior under worst-case bounded perturbations of the visual tokens. It trains only low-rank adapters, uses no attack-specific data, and does not depend on keeping any parameter secret.
Concretely, let $v$ be the visual tokens of an input, $t$ its accompanying text, and $a$ a text-only rendering of the same request, meaning the image content described in words. Let $p_\phi$ be the adapted model with trainable adapter weights $\phi$, $p_0$ the frozen original model, and $y_{1:k}$ the first $k$ response tokens. We train
$$\min_\phi \; \mathbb{E}_{(v,t,a)} \Big[ \max_{\|\Delta\| \le \rho} \mathrm{KL}\big(p_0(y_{1:k} \mid a) \,\|\, p_\phi(y_{1:k} \mid v+\Delta, t)\big) \Big],$$
where $\Delta$ is a perturbation of the visual tokens with norm budget $\rho$, and the inner maximization is a few steps of projected gradient ascent. The same term is applied to harmful and benign pairs. On harmful pairs the anchor $p_0(\cdot \mid a)$ is a refusal. On benign pairs it is a helpful answer, so the adversary is also searching for perturbations that push benign inputs into refusal. This symmetry is the proposal's direct answer to over-refusal. USB shows how severe over-refusal can be: Claude 3.5 Sonnet had the lowest attack success rate there, 15.57%, while refusing 25.82% of harmless prompts. Restricting the loss to the first $k$ tokens keeps the anchor cheap. It also targets the point where refusals and compliances diverge.
Research Program
The work has three parts.
Training data that excludes the attacks we test on. We take harmful and matched benign text instructions and create multimodal versions in one way only. The key object is shown as a natural photograph, and the text refers to it ("this"). The text-only anchor spells the object out. No typographic images, no diffusion-amplified images, and no pixel-attacked images appear in training. Every evaluation attack family is therefore out of distribution by construction.
The inner attacker. The attacker is $K$-step projected gradient ascent on the visual tokens at the projector output. We sweep $K$ over small values and sweep the budget $\rho$. We report wall-clock and GPU-hours against a pixel-space inner loop and against ADPO. The savings claim is deliberately modest: the language-model backward pass remains, and the gains come from skipping the vision encoder and from needing few steps. We measure the savings rather than assume them.
Analysis of whether the mechanism is what we claim. We compute a refusal direction in the backbone's hidden states from text-only harmful versus harmless prompts. We then check two things. First, whether each unseen attack family lowers the projection onto this direction in the base model. Second, whether the adapted model restores it. If some attack family does not act through this suppression, we expect the defense to fail on it. That would be an informative negative result about which attacks share the chokepoint.
Models are LLaVA-1.5-7B, where ADPO reports numbers, and Qwen2-VL-7B. Training uses low-rank adapters on the language backbone and projector, and the vision encoder stays frozen.
Baselines
The naive reference points are the unmodified model and plain supervised safety fine-tuning on our clean training pairs. The latter is the SFT condition that ADPO reports reaching only 24% safe responses on LLaVA-1.5 under the white-box MMPGDBlank attack, against 67% for standard DPO.
ADPO. The strongest existing competitor is ADPO. It trains an adversarially robust reference model with multi-step projected gradient descent in pixel or latent space and then runs preference optimization on adversarial examples generated on the fly. It reports a 99.5% safe response rate on LLaVA-1.5 under MMPGDBlank. It is the only method in the file that combines adversarial training with generative multimodal safety, so it is the one to beat on both cost and generalization. For a fair comparison we train it on our same instruction pool, with its latent-space variant included, at matched adapter capacity. Both methods get the same number of hyperparameter trials. We would not expect to beat 99.5% on its home benchmark. The proposal counts as winning if it comes within a few points there at a clearly lower measured training cost, and if it has lower attack success than ADPO on the held-out typographic, composite, and safe-image/safe-text families.
Contrastive Harmlessness LoRA. This defense from HADES fine-tunes on harmful/harmless multimodal pairs. It reduced LLaVA-1.5's HADES attack success from over 90% to 5.07%. We train it on its own curated data, which is its intended setting, and evaluate it on every benchmark other than HADES. This shows how much of its gain is in-distribution.
Training-free prompts. We include the MM-SafetyBench safety prompt, which lowered LLaVA-1.5-7B attack success on the Tiny set from 77.33% to 15.68%. We also include the Adversarial Possibility prompt of Bhagwatkar et al., which raised LLaVA-7B's COCO accuracy under Auto-PGD from 12.54% to 91.46%. Both are evaluated under adaptive attacks that include the prompt in the attacker's objective.
Ablations. Two ablations isolate the proposal's ingredients. The first removes the adversary ($\Delta = 0$), which tests whether text anchoring alone suffices. The second keeps the anchor loss but moves the inner attacker to pixel space, which tests whether the latent chokepoint matters for generalization or only for cost. Both use identical data and training steps.
Why not just…?
Why not just use a safety system prompt? It costs nothing, and on MM-SafetyBench it cut attack success from 77.33% to 15.68%, so it is a real threat on that benchmark. Against it, FigStep reports that system-prompt changes were largely ineffective against typographic prompts. Bhagwatkar et al. also note that a fixed prompt can simply be folded into a white-box attacker's objective. Our experiments run both prompts against the full held-out suite, with and without adaptive attacks. If the prompts match us everywhere, including under adaptive attack, the training is not worth doing and we will say so.
Why not just caption or read the image first and send only text to the aligned model? This is the most direct way to reuse text alignment. It plausibly handles typographic attacks, since reading the text converts them back into text requests the model already refuses. It has three weaknesses. It discards visual detail that benign tasks need. The captioner is itself a model open to pixel attacks. And safe-image/safe-text compositions may survive captioning, because the harm lies in the combination rather than in either part. We run a caption-then-answer pipeline using the same backbone and measure it on OCRBench and MM-Vet as well as on the safety suites. If it matches our safety numbers at comparable utility, that is a finding the proposal must report.
Why not just run ADPO, or ablate the text-reading heads? ADPO trains against the attack it is evaluated on. Its own results do not test transfer to unseen families, and its online pixel-space loop is the cost we aim to cut. The head-to-head comparison above settles which is better. Dyslexify does improve typographic robustness, by 22.06% on ImageNet-100-typo. It works by making CLIP unable to read, which conflicts with the OCR abilities VLMs need. It also offers no protection against pixel noise. Our OCRBench check determines whether anchoring avoids that trade-off.
Proposed Experiments
The decisive test is a held-out-family evaluation. After training only on natural-photo pairs, we measure attack success on five families:
- typographic prompts: SafeBench and SafeBench-Tiny from FigStep, where the undefended open-source average is 82.50%;
- composite scenes: MM-SafetyBench;
- diffusion-amplified images: the HADES benchmark, 90.26% on undefended LLaVA-1.5;
- safe-image/safe-text compositions: the USB-Hard split of USB, where open-source models exceed 80%;
- white-box pixel attacks: VisualAdv and MMPGDBlank as used by ADPO.
Harmfulness is judged by the protocol each benchmark specifies. Utility is measured on MMStar, OCRBench, MM-Vet, and LLaVABench. Over-refusal is the refusal rate on USB's harmless prompts.
The adaptive check matters because the adapter weights are public in our threat model. We run full white-box pixel-space attacks, targeting affirmative completions, through the entire defended model, at the $\ell_\infty$ budgets of 1/255 and 4/255 used by Schlarmann and Hein. A defense that only holds against latent-space attackers, or against attackers unaware of the adapters, fails this check.
The proposal is supported if three things hold. Out-of-family attack success falls well below ADPO's and Contrastive LoRA's on at least three of the four non-pixel families. OCRBench and over-refusal stay close to the base model. And the no-adversary ablation is clearly worse than the full method on the held-out families.
The proposal is refuted if the pixel-space-anchor ablation generalizes as well as the latent version. In that case the chokepoint argument is wrong and only the cost claim survives. It is also refuted if the refusal-direction analysis shows that the unseen families do not act through refusal suppression, since the generalization story would then have no mechanism.
References
- $\mathsf{FigStep}$ : Jailbreaking Large Vision-Language Models via Typographic Visual Prompts
- MM-SafetyBench: A Benchmark for Safety Evaluation of Multimodal Large Language Models
- USB: A Comprehensive and Unified Safety Evaluation Benchmark for Multimodal Large Language Models
- On the Adversarial Robustness of Multi-Modal Foundation Models
- Images are Achilles' Heel of Alignment: Exploiting Visual Vulnerabilities for Jailbreaking Multimodal Large Language Models
- Adversary-Aware DPO: Enhancing Safety Alignment in Vision Language Models via Adversarial Training
- Improving Adversarial Robustness in Vision-Language Models with Architecture and Prompt Design
- Dyslexify: A Mechanistic Defense Against Typographic Attacks in CLIP