Teaching the Projector to Answer Like Text

Vision-language models are built by bolting an image encoder onto a safety-tuned language model. A small "projector" network sits between them and translates image features into the vectors the language model reads as words. The language model, given text alone, refuses most harmful requests. Given the same request routed through an image, it often complies. On LLaVA-1.5-7B, MM-SafetyBench reports an attack success rate for illegal-activity queries of 5.25% as plain text and 79.38% when the query is rendered as typography in an image.

We propose to close this gap by training only the projector, with no refusal labels. The training signal is the model's own text-only behavior. For each unlabeled image and prompt, we caption the image, including any visible text, with the original frozen model. We then train the projector so that the model's first response tokens given the image match its first response tokens given the caption. This match must hold even after a cheap one-step worst-case perturbation of the image.

In two sentences: the language model already knows when to refuse, and it fails only because image tokens do not "sound like" the harmful text they encode. So we teach the projector to make images sound like their descriptions, and keep that true under small image perturbations, using one gradient step per batch instead of a full inner attack loop.

Core Claim

Robustness to visual jailbreaks can be distilled from the language model's own text safety. It does not need human-written refusal targets, and it does not need multi-step adversarial search during training. Three properties make this approach distinctive.

First, the target is a text-anchored virtual label. Classic virtual adversarial training (VAT) uses the model's own clean prediction as the label. Here the label is the frozen model's prediction on the caption.

Second, the regularized component is the projector. IPGA showed the projector is where attacks gain the most leverage, but it proposed no defense.

Third, the language model's weights never change. Text-only safety is therefore preserved by construction. This sidesteps the forgetting of text safety during parameter-efficient multimodal tuning that VLGuard reports.

Formally, write $E$ for the frozen image encoder and $P_\phi$ for the projector with parameters $\phi$. Write $p_\theta$ for the frozen language model's next-token distribution, $x$ for the image, $t$ for the prompt, and $c(x)$ for the caption produced by the original model. Let $y$ be the response the frozen model generates from $(c(x), t)$. We minimize

$$\mathcal{L}(\phi)=\mathbb{E}_{x,t}\Big[\sum_{j\le k}\mathrm{KL}\big(p_\theta(\cdot\mid c(x),t,y_{<j})\,\|\,p_\theta(\cdot\mid P_\phi(E(x+\delta^\star)),t,y_{<j})\big)\Big]+\lambda\,\mathcal{L}_{\text{keep}}(\phi).$$

The pieces of this objective are defined as follows:

  • $k$ is a short prefix length, around 16 tokens, because the decision to refuse or comply is made early in a response.
  • $\delta^\star$ is a perturbation within an $L_\infty$ budget. It is found by one power-iteration step on this same divergence, as in VAT, which costs about one extra forward and backward pass.
  • $\mathcal{L}_{\text{keep}}$ is the divergence from the original model's outputs on clean benign images over the full response. It protects visual detail that captions lose.

Relation to Prior Work

The work closest to ours is ProEAT. Like us, it trains only the projector. Unlike us, it aligns perturbed inputs to fixed refusal strings, generates perturbations with multi-step PGD (projected gradient descent, an iterative attack) and GCG (a text-suffix attack), and takes roughly 15 hours. We remove both the refusal labels and the iterative attack loop.

The second closest is Sim-CLIP+. It is label-agnostic, but it hardens only the image encoder with a clean-versus-perturbed similarity loss. Hardening the encoder cannot address jailbreaks that succeed with blank or noise images, which JailBreakV-28K and Images are Achilles' Heel of Alignment document. Our anchor targets exactly those cases: the caption of a noise image reduces to the bare harmful text, which the language model refuses.

Three lines of work show that label-free robustness is feasible but constrain how we design it:

  • RoCL, ProFeAT and ASTrA show self-supervised adversarial training works on single-modality encoders. They are also expensive, which is why we use a one-step adversary.
  • VARMAT shows first-order, single-step sensitivity penalties can replace PGD in multimodal classifiers. It depends on task labels and was never tested on generative models.
  • Deep Contractive Networks and gradient regularization motivate penalizing local sensitivity instead of searching for attacks.

Two inference-time defenses show the cost of not changing the model's weights. RESTA votes over many noisy forward passes per generated token. VLMGuard is an external detector. Neither hardens the model against white-box gradients.

ADPO hardens weights through adversarial preference training. It needs both labeled preference pairs and a heavy adversarial reference model.

Research Program

Data. We assemble unlabeled image–prompt pairs. One part is benign instruction data. The other part is the images and prompts of the VLGuard training split, discarding its responses and its harmful/benign tags. This keeps harmful-intent inputs present in the training distribution without using any annotation.

Anchors. Captions and anchor responses are generated once, offline, by the original model. The caption prompt asks it to "describe the image and transcribe any visible text."

Training. Training updates only the projector. Each step uses one VAT power iteration through the frozen encoder.

Analysis. We will measure how far the projector's outputs move under attack, in the language model's own output distribution, before and after training. This tests directly whether the projector stops amplifying the perturbation directions that IPGA exploits.

Models. The main model is LLaVA-1.5-7B. We use Bunny-v1.0-4B where we compare directly with ProEAT.

Baselines

The do-nothing reference is the unmodified model. It also gets a defensive system prompt, which MM-SafetyBench reports cuts LLaVA-1.5-7B's attack success rate on its tiny split from 77.33% to 15.68%. FigStep reports this prompt is largely ineffective against typographic attacks.

The strongest existing method is ProEAT, retrained on the same base model and the same image–prompt pool, with its refusal targets and PGD/GCG inner loop. ProEAT reports a weighted attack success rate of 0.002 on Bunny-v1.0-4B. It also reports an average margin of +34% over prior baselines and about 15 hours of training. Beating it means matching its white-box and black-box attack success rates within noise, at no more than one-third of its GPU-hours and with equal or better clean utility. Compute is logged identically for both methods, and both receive the same hyperparameter search budget.

VLGuard post-hoc supervised fine-tuning is the strongest label-dependent cheap method. It reports FigStep attack success dropping from over 90% to 0%. ADPO reports that supervised safety fine-tuning reaches only a 24% safe-response rate under the white-box MMPGDBlank attack on LLaVA-1.5. We run VLGuard on the same model, restricted to projector-only and to LoRA variants.

Sim-CLIP+ is run as a drop-in encoder swap. Its reported ImgJP attack success rates on LLaVA are 15.0% with a Llama-2-13B backbone and 19.3% with Vicuna-7B. RESTA is the inference-time reference. It reports reducing JailBreakV-28K attack success from 50.13% to 25.93%, with ScienceQA accuracy falling from 64.07% to 61.42%.

Three ablations isolate the key ingredient:

  • Self-anchor. The caption anchor is replaced by the model's own clean-image output, which is pure VAT on the projector.
  • PGD inner loop. The one-step adversary is replaced by a 10-step PGD adversary.
  • Prefix length. The prefix length $k$ is set to 4, 16 and the full response.

Why not just…?

Why not just fine-tune on VLGuard? VLGuard is the real threat on black-box and typographic attacks. We expect to merely tie it there. The reason to do this work is white-box robustness, where ADPO's evidence shows supervised fine-tuning collapses. Our white-box PGD and IPGA-style evaluations settle this question. If our method does not beat VLGuard under white-box attack, it has no reason to exist.

Why not just caption first and answer from the caption at inference? This pipeline inherits text safety directly, and it is the most honest competitor to our anchor idea. It roughly doubles inference cost and discards visual detail. It is also still attackable, because an attacker can perturb the image to steer the caption. We include it as a baseline, scoring both its utility on ScienceQA, COCO and OKVQA and its attack success under end-to-end white-box attack through the captioning step.

Why not just harden the encoder (Sim-CLIP+)? Its own reported HADES attack success rates stay at 64–74%. Blank-image and noise-image jailbreaks need no encoder perturbation at all. We report results split by JailBreakV-28K image type to show where hardening the encoder alone stops helping.

Why not just use self-anchored VAT, without captions? Smoothing toward the clean-image output preserves whatever the model already says. On typographic or blank-image inputs the model already complies, so smoothing toward that output keeps it complying. The self-anchor ablation should match our method on small-perturbation attacks and fail on FigStep and MM-SafetyBench typography. If it does not fail there, the caption anchor is unnecessary and we will say so.

Proposed Experiments

Black-box safety. We evaluate on four benchmarks:

  • MM-SafetyBench, all three attack styles, reporting attack success rate and refusal rate.
  • JailBreakV-28K, split by image type.
  • FigStep's SafeBench.
  • OmniSafeBench-MM through its toolbox, scored with its harmfulness/intent-alignment/detail protocol.

White-box safety. We evaluate on VisualAdv, ImgJP and HADES. We add adaptive attacks that run PGD end-to-end through the encoder, projector and language model, plus an IPGA-style attack aimed at the projector. Gradient-free and transfer attacks check that robustness is not an artifact of masked gradients.

Utility. We measure ScienceQA, COCO captioning and OKVQA, and compare refusal rates on benign VLGuard test prompts to catch over-refusal.

Decisive checks. The first decisive check is whether the caption-anchored, one-step projector matches ProEAT's white-box attack success rate at one-third of the compute. The second is whether the self-anchor ablation fails specifically on typographic and blank-image attacks. Together these show that the anchor, not the smoothing alone, is what closes the safety gap.