Multimodal language models are chat models that accept images as well as text. They are much less safe than their text-only parents. A request the model refuses as text is often answered when it arrives as text rendered inside an image. Even a blank image attached to a harmful text query weakens refusal. Models can also be "backdoored": during fine-tuning, an attacker plants a hidden visual pattern (a trigger) that makes the model produce an attacker-chosen output. This note proposes one cheap repair for both problems. Take a few hundred unlabeled images, describe each in words with an independent captioning model, and train the vision-language model so that its response to the image matches its own response to the written description. The text pathway already carries the safety training the image pathway lacks. A visual trigger that a captioner would not mention cannot steer the model if the model is forced to behave as it would on the caption.
The approach matters because current repairs are costly or narrow. One family is adversarial training: the model is trained on worst-case inputs that an inner optimization loop finds, typically projected gradient descent (PGD), meaning repeated gradient steps on the input. This multiplies training cost, and the resulting robustness tends to hold only for the kind of perturbation used in training. Text-only safety unlearning is fast, but it fails when the danger lives only in the picture. Test-time backdoor purification adds latency to every query and targets localized patches. Self-distillation from a caption needs no inner attack loop, no harmful labels, and no extra inference cost. Its invariance target is defined by meaning ("what is in this image") rather than by a perturbation budget, which is the property that should let it generalize beyond the attacks seen in training.
Core Claim
The multimodal safety gap and visual backdoors share one measurable signature: the model's behavior on an image diverges from its behavior on a faithful verbal description of that image. A backdoor is a divergence the attacker installed. The safety gap is a divergence that visual instruction tuning left behind. Both can therefore be closed with the tool data-efficient backdoor defenses already use, which is distilling a trusted teacher's internal behavior into the compromised student on a small clean set. The new part is the choice of teacher. Here it is the same frozen model reading a caption instead of the pixels. This makes the multimodal model inherit text-domain alignment on visual inputs while ignoring visual content that has no verbal meaning.
The claim is falsifiable on the benchmark designed to break text-space shortcuts. VLSBench removes harmful keywords from the text, so the hazard is visible only in the image. Our teacher sees the hazard because the caption verbalizes it. Text-only unlearning never sees it.
How This Differs from the Closest Work
Three prior lines motivate the proposal and supply its method.
Neural Attention Distillation erases backdoors from image classifiers. It briefly fine-tunes a copy of the model on 5% clean data to make a teacher, then distills the teacher's intermediate attention maps back into the backdoored student. We reuse the distill-from-a-cleaner-view principle, but our teacher is a different input view of the same frozen model rather than a fine-tuned copy, and our setting is a generative multimodal model.
Backdoor Unlearning by Linear Task Decomposition subtracts an estimated "trigger direction" in weight space. It needs triggered examples or a trigger reverse-engineering tool. Our method needs neither, because the caption filters out non-semantic triggers by construction.
PurMM and TCAP detect backdoors in multimodal models through attention anomalies. PurMM's finding that trigger-driven "attention hijacking" concentrates in deep layers guides where we place trainable parameters. Neither method was evaluated against jailbreaks.
Two other papers constrain the design.
Cross-Modal Safety Alignment: Is textual unlearning all you need? argues that all inputs pass through the language model, so aligning text suffices. VLSBench refutes this on leakless pairs: textual SFT reaches a 13.99% safe rate on LLaVA-v1.5-7B, against 21.26% for multimodal SFT. We accept the bottleneck intuition but reject its conclusion. The language model can only apply its alignment if the visual hazard reaches it in a form that alignment recognizes, and caption distillation forces exactly that.
E²AT is the closest efficient multimodal defense. It runs PGD adversarial training on the projector (the small module mapping image features into the language model) jointly with language-model optimization. Our method has no inner attack loop, and it also targets backdoors and leakless threats, which E²AT does not test.
Method
Let $x$ be an image, $q$ a text query, $c(x)$ a caption from an independent captioning model, and $p_\theta$ the vision-language model with trainable parameters $\theta$. These are the projector plus low-rank adapters on the deeper half of the language model's layers. Let $\theta_0$ be the original weights, which stay frozen as the teacher. Let $y$ be the teacher's own greedy response to the text-only input $(c(x), q)$, and $y_{<t}$ its first $t-1$ tokens. Training minimizes
$$\mathcal{L}(\theta)=\sum_{t\le K}\mathrm{KL}\big(p_{\theta_0}(\cdot\mid c(x),q,y_{<t})\,\|\,p_\theta(\cdot\mid x,q,y_{<t})\big)+\mu\,\mathrm{KL}\big(p_{\theta_0}(\cdot\mid x,q_b)\,\|\,p_\theta(\cdot\mid x,q_b)\big).$$
The first term pulls the image-conditioned response toward the caption-conditioned one over the first $K$ response tokens, where refusal or compliance, or a backdoored answer, is decided. The second term, weighted by $\mu$, keeps the model unchanged on benign queries $q_b$ that need fine visual detail a caption would omit. This guards against the obvious failure of distilling a weaker, caption-only view.
The training set is a few hundred natural images, each paired with a mix of benign queries and harmful text queries. The pool is drawn so that it is disjoint in category from every evaluation benchmark. No safety labels are needed, because the teacher's behavior is the label.
Each training step costs two forward passes and one backward pass. A PGD loop costs many more. We will report wall-clock GPU hours directly rather than assert savings.
Research Program
The work has three stages.
The first stage tests whether the signature exists. On LLaVA-v1.5-7B, we measure how far apart the image-conditioned and caption-conditioned outputs are, in KL divergence and in refusal agreement. We do this on JailBreakV-28K, MM-SafetyBench typography images, SIUO and VLSBench, and on PurMM-style backdoored checkpoints with BadNet, Blend and SIG triggers. If backdoored and jailbroken inputs do not show markedly higher divergence than benign ones, the premise fails and we stop there.
The second stage trains the distillation and maps which ingredients matter: projector-only versus deep-layer adapters, the token horizon $K$, and caption quality, comparing a strong captioner, a weak captioner, and the model's own captions.
The third stage stress-tests generalization and adaptive attacks. This includes TA-SPA semantic perturbations and white-box PGD on the defended model. The white-box attack is the regime where this method has no built-in advantage over adversarial training.
Baselines
The naive reference is the undefended model. For LLaVA on JailBreakV-28K, E²AT reports a weighted attack success rate (ASR) of 0.452. We re-measure this with the same Llama-Guard judge used for every method. The zero-cost reference is a prepended safety prompt. MM-SafetyBench reports that it reduces LLaVA-1.5-7B ASR on its tiny split from 77.33% to 15.68%.
The strongest existing efficient jailbreak defense is E²AT. It reports a weighted ASR of 0.054 on JailBreakV-28K with LLaVA. It is the right competitor because it also trains only light components and also claims to keep clean performance. We will run it from its described recipe on the same LLaVA-v1.5-7B checkpoint, with matched trainable parameter count and an equal hyperparameter-search budget, and report its GPU hours next to ours. Beating it means matching its JailBreakV ASR within noise, while clearly exceeding it on VLSBench and SIUO safe rates, at a fraction of its training compute. Winning on cost alone with worse leakless safety would not count.
The two text-side competitors are textual unlearning from Cross-Modal Safety Alignment and multimodal SFT. On VLSBench, VLSBench reports 13.99% and 21.26% safe rate respectively for LLaVA-v1.5-7B. We rerun both with the same base model and with their published data sizes. Beating them means a VLSBench safe rate above 21.26% under a matched or smaller data budget.
For backdoors, the reference is PurMM. It reports that LLaVA-v1.5-7B ScienceQA ASR falls from 99.55% to 0.84%. We also include TCAP as a training-data filter. We run both on identical poisoned checkpoints. Success means comparable ASR reduction with no per-query test-time cost and clean ScienceQA accuracy within one point.
For inference-time captioning, no reported result was found on Lacuna for caption-then-respond pipelines on these benchmarks, so we build one ourselves.
Two ablations isolate the key ingredient. The first replaces the caption teacher with the same model seeing the original image under a safety prompt, which removes verbalization. The second distills from a caption of a different, mismatched image. If either ablation matches the full method, the gains come from regularization rather than from the verbal view.
Why not just…?
Why not just caption every image at inference and let the text model decide? This is the most serious threat. It probably matches our safety numbers, because it is our teacher. It pays for a captioner plus a longer prompt on every query, and it discards visual detail on benign tasks unless one also routes between the two views. We will build this pipeline and report safety, clean accuracy and latency side by side. If it wins on safety and the latency is tolerable, the honest contribution narrows to amortizing that pipeline into the weights. That result is still useful, but it is smaller than the core claim.
Why not just do text-only unlearning, which is six times cheaper than multimodal SFT? VLSBench shows that its success depends on harmful keywords leaking into the text: textual SFT reaches only 13.99% safe rate on leakless pairs. Our training costs the same order as text unlearning, since it uses no generated images and no attack loop, but its teacher sees the visual hazard through the caption. The VLSBench comparison settles this directly.
Why not just use test-time token masking like PurMM? It is training-free and very effective on patch triggers. It clusters attention maps for every input, and its design assumes a localized anomalous region. That assumption does not hold for typographic jailbreaks or for distributed perturbations like TA-SPA, which reaches 52.3% G-ASR against commercial APIs. We report PurMM's ASR on those attacks and its per-query latency alongside our backdoor results.
Why not just run cheap adversarial training on the projector? That is E²AT, and against white-box pixel attacks it may well beat us, because our training never sees adversarial pixels. We test this explicitly. We also test a variant that adds a single random-noise or one-step perturbation to the student's image while the caption teacher stays fixed, to see whether that closes the gap without returning to multi-step PGD.
Proposed Experiments and Decisive Checks
All main experiments use LLaVA-v1.5-7B, with Bunny-v1.0-4B as a second model for jailbreaks and InternVL2.5-8B for backdoors, matching the models used by E²AT and PurMM.
Safety is measured as weighted ASR on JailBreakV-28K under all four image types, ASR and refusal rate on MM-SafetyBench (SD, Typography and combined images), safe rate on SIUO's 269 cases and VLSBench's 2,241 pairs, and transfer ASR under TA-SPA. Backdoor removal is measured as ASR and clean accuracy on ScienceQA and DocVQA under BadNet, Blend, SIG and WaNet. Over-refusal is tracked as clean accuracy on the same task sets. Cost is measured in training GPU hours and per-query latency.
Four outcomes decide the proposal. First, the full method must exceed 21.26% safe rate on VLSBench without any multimodal safety labels; otherwise the verbal teacher adds nothing over SFT. Second, the mismatched-caption ablation must lose most of the gain. Third, backdoor ASR must fall to PurMM's range using only a few hundred unlabeled clean images and no trigger knowledge. Fourth, gains must hold on harm categories excluded from training, which is the direct test of whether caption-based invariance escapes the adversarial generalization gap that perturbation-specific defenses show. We also report one negative-control check: an adaptive attacker who optimizes images against the defended model. That attack shows how much of the remaining risk comes from pixel-level attacks, the regime this method was never meant to cover.
References
- VLSBench: Unveiling Visual Leakage in Multimodal Safety
- Neural Attention Distillation: Erasing Backdoor Triggers from Deep Neural Networks
- Backdoor Unlearning By Linear Task Decomposition
- PurMM: Attention-Guided Test-Time Backdoor Purification in Multimodal Large Language Models
- TCAP: Tri-Component Attention Profiling for Unsupervised Backdoor Detection in MLLM Fine-Tuning
- Cross-Modal Safety Alignment: Is textual unlearning all you need?
- E$^2$AT: Multimodal Jailbreak Defense via Dynamic Joint Optimization for Multimodal Large Language Models
- MM-SafetyBench: A Benchmark for Safety Evaluation of Multimodal Large Language Models
- Text-Anchored Semantic Perturbations for Transferable Jailbreak Attacks on Multimodal Large Language Models