Removing a hidden backdoor from a model almost always costs some ordinary performance. A backdoor, or trojan, is a planted behavior: the model acts normally until it sees a secret trigger, then does what the attacker wants. Every current repair method trades removal strength against clean-data quality through a hand-set weight on a "forget the backdoor" term. In vision-language models, which read an image and a text prompt and generate text, one weight is the wrong control. "Ordinary performance" there covers several things at once: caption quality, answers to text-only questions, and whether the model starts refusing harmless image-and-text requests. A single weight cannot hold all of these in place, and a weight tuned on one model does not carry over to a larger one.
We propose to recast backdoor removal as constrained optimization. The goal is to drive trigger-conditioned behavior toward zero subject to explicit budgets, each written in the units practitioners care about: a caption-score drop of at most one point, and a benign-refusal increase of at most two points on harmless image-and-text pairs. Each budget gets its own automatically adjusted multiplier. The adversary that searches for triggers during repair searches jointly over the image and the text, so that it can find triggers that live only in the combination of the two. The central claim is testable. Budgets stated in metric units should transfer across model scale and attack type, where tuned loss weights do not, and several budgets can be met at once where a single weight must sacrifice one to meet another.
Core Claim
Each of the four directions plays a distinct role here.
Backdoor attacks supply the threat. VL-Trojan poisons the instruction-tuning stage of autoregressive vision-language models such as OpenFlamingo. It uses an image-patch trigger optimized so that even a frozen image encoder maps it to a distinct cluster. Poisoning about 0.5% of the data gives above 99% attack success, and clean caption scores stay virtually unchanged. That paper proposes no defense. It also reports that 9B models memorize the backdoor more readily than 3B models.
The robustness-accuracy trade-off supplies the cost we must control. Pruning, unlearning and repair fine-tuning all damage clean features entangled with the trigger:
- Fine-Pruning and Adversarial Neuron Pruning both show that aggressive pruning hurts clean accuracy.
- For CLIP-style contrastive models, which learn to match images with captions, UBT drops clean accuracy from 62.58% to 60.81% under a Blended attack.
- ABD pays an average 1.73% drop.
Hyperparameter sensitivity supplies the diagnosis.
- Obliviate says its neuron-amplification term needs careful manual monitoring, because too much amplification breaks convergence.
- The verified-robustness literature finds the same brittleness. Loss weights fail to transfer across architectures in interval bound propagation and in expressive losses.
The multimodal safety gap supplies the extra utility dimension and the hard case. USB shows the safest model it tested, Claude 3.5 Sonnet, also refuses 25.82% of harmless queries. A defense that "removes" a harmful-output backdoor by making the model refuse more has therefore not succeeded. SIUO shows that harm can emerge from an image and a text that are each benign. This suggests a composite trigger, meaning a trigger present only when a particular benign image and a benign phrase occur together. Token-level unlearning like UBT, and image-side adversarial repair like ABD, both miss such triggers by construction.
The method comes from DALE. DALE minimizes robust loss subject to a single bound on clean loss and updates a dual variable automatically. On CIFAR-10 with ResNet-18 it holds above 85% clean accuracy at above 50% PGD-20 robustness, where TRADES and MART fall below 80% clean accuracy. DALE has only one constraint, is unimodal, and handles only norm-bounded perturbations, not backdoors. The inner search that plants synthetic triggers comes from I-BAU. I-BAU unlearns backdoors with about 100 clean images by solving a min-max problem through implicit hypergradients, but only for additive image perturbations on classifiers.
What is new here is the combination of three pieces:
- several budgets, one per utility slice, each with its own multiplier;
- a joint image-and-text inner adversary;
- a generative vision-language setting where over-refusal counts as a failure.
Research Program
Let $\theta$ be the model parameters and let $(\delta, \epsilon)$ be a candidate trigger: $\delta$ is a bounded image perturbation or patch, and $\epsilon$ is a small perturbation to text embeddings. $L_{\text{bd}}(\theta; \delta, \epsilon)$ is the backdoor loss. It is the likelihood of the model producing attacker-like behavior when the candidate trigger is present, measured on a small trusted clean set. $L_k(\theta)$ is the clean loss on utility slice $k$, and $\rho_k$ is that slice's budget, expressed as a tolerated change from the poisoned model's starting value. We solve
$$\min_{\theta} \; \max_{\delta, \epsilon} \; L_{\text{bd}}(\theta; \delta, \epsilon) \quad \text{subject to} \quad L_k(\theta) - L_k(\theta_0) \le \rho_k \;\; \text{for } k = 1, \dots, K,$$
where $\theta_0$ is the poisoned starting model. We solve it by alternating gradient steps on $\theta$ with projected ascent on the multipliers:
$$\nu_k \leftarrow \max\big(0,\; \nu_k + \eta\,[L_k(\theta) - L_k(\theta_0) - \rho_k]\big).$$
Here $\nu_k$ weights slice $k$ in the Lagrangian, and $\eta$ is a dual step size. We will check that results are insensitive to $\eta$.
We use four slices ($K = 4$):
- caption and question-answering likelihood on clean images;
- text-only instruction following;
- refusal-free responses on the harmless image-and-text pairs from USB's safe-image/safe-text split;
- retained refusals on genuinely harmful requests, so that repair does not erode the model's original safety alignment.
The budgets $\rho_k$ are converted from metric targets using a short calibration run on the trusted set.
We depart from DALE in two ways. First, we replace its Langevin sampling with a few projected ascent steps, following I-BAU, to control cost; one ablation restores sampling. Second, the inner search runs over $\delta$ and $\epsilon$ jointly, which lets it find triggers that exist only in the combination.
The repair itself is a low-rank adapter update on the language side and the vision-to-language connector. This matches where VL-Trojan's backdoor lives: the image encoder is frozen and the trainable connector learns the malicious mapping.
We will also measure how the learned multipliers split across slices. Frank et al. find that vision-language transformers lean on vision to model language far more than the reverse. That asymmetry predicts the image-dependent slices should need larger multipliers. If they do, it would be a mechanistic explanation for why one scalar weight fails.
Baselines
The naive reference points are the poisoned model left untouched and the same adapter fine-tuned on the trusted clean set with no removal term. BackdoorMBTI found plain fine-tuning to be a universal baseline across image, text and audio that lowers attack success without severe clean loss. It is therefore a serious competitor, not a strawman. It gets the same data, adapter rank and optimizer steps as our method.
The strongest existing method is a fixed-weight version of our own objective. It uses the same joint I-BAU-style adversary and the same data, with one scalar $\lambda$ on the removal term. We give it a grid search whose total GPU hours equal our method plus its calibration run, and select on a validation split of the clean slices. This isolates the question that matters: whether automatic per-slice multipliers beat a well-tuned single weight at equal compute.
We also port three published defenses to the new setting:
- Obliviate's amplification and attention regularizers, applied to the adapter;
- HADES's Contrastive Harmlessness LoRA;
- ABD-style image-side adversarial repair.
For reference, these papers report the following, each in its own setting:
- Obliviate: 83.6% average attack-success reduction with a 0.78% clean drop, on text classifiers.
- HADES LoRA: LLaVA-1.5 jailbreak success reduced from above 90% to 5.07%.
- ABD: 1.73% average clean drop, on CLIP.
- VL-Trojan (the attack): above 99% success on COCO and Flickr-30K with CIDEr virtually unchanged.
We found no reported result on Lacuna for any defense against VL-Trojan-poisoned autoregressive models, so all defense numbers in this setting will be produced by us under shared protocols.
Two ablations isolate the key ingredients. The first keeps the constraints but collapses them to a single aggregate clean-loss budget, which is DALE's structure. The second keeps the per-slice multipliers but restricts the adversary to the image side.
Beating the baselines means reaching lower attack success than the equal-compute grid while meeting all four budgets simultaneously on held-out data. It also means doing so on the 9B model with budgets copied unchanged from the 3B model.
Why not just…?
Why not just grid-search $\lambda$ on a clean validation set? For one model and one utility metric, this may well work. That is why it is our main baseline at matched compute, not a dismissed alternative. Our claim is narrower and falsifiable in two parts. First, with four slices, a single $\lambda$ traces a one-dimensional curve that may never enter the region where all budgets hold. Second, a $\lambda$ chosen on OpenFlamingo-3B will miss its budgets on 9B, because VL-Trojan shows the larger model memorizes the trigger more strongly and so needs a different removal pressure. If the grid-selected $\lambda$ transfers across scale and meets all slices, our contribution shrinks to convenience, and we will say so.
Why not just fine-tune on clean data? BackdoorMBTI's evidence favors this in unimodal settings. However, Fine-Pruning argues that backdoors sit in spare capacity that clean data rarely activates, so clean gradients may never reach them. The prediction is testable: plain fine-tuning should leave composite-trigger success high, because nothing in clean data exercises those pathways. If it removes composite triggers as well as our method does, the joint adversary is unnecessary.
Why not just filter triggered inputs at runtime? The detector in Architectural Backdoors in VLM Supply Chains flags drift in hidden states rather than repairing the model. Its authors also leave open how to set detection thresholds against false alarms on unusual benign inputs, which is the same budgeting problem in another form. Baseline Defenses for aligned LLMs shows that input filters misflag atypical benign queries. We will run this detector at thresholds matched to our benign-refusal budget and compare the attack success left over.
Why not just apply safety fine-tuning such as HADES's LoRA or HarmBench's R2D2? These teach refusal, and USB's refusal paradox predicts they will buy lower attack success with over-refusal on harmless pairs. Our benign-refusal slice measures exactly this cost.
Proposed Experiments
We poison OpenFlamingo-3B and 9B, plus LLaVA-1.5, on the MIMIC-IT LADD subset using three triggers:
- VL-Trojan's patch trigger;
- VL-Trojan's character-level text trigger;
- a new composite trigger that pairs a benign scene category with a benign phrase, built following SIUO's safe-plus-safe construction.
Attack targets include both a fixed malicious caption and harmful-instruction compliance.
We measure four things:
- attack success on triggered COCO and Flickr-30K inputs;
- CIDEr on clean captions;
- over-refusal on USB's safe-image/safe-text split, from the cgjacklin/USB dataset;
- retained safety on SIUO and HADES, scored with HarmBench's classifier.
HarmBench shows that generation length alone can shift attack success by up to 30%, so we fix generation length across all conditions.
Three checks decide the question.
Transfer. Set $\rho_k$ or $\lambda$ on 3B and apply both unchanged to 9B and to LLaVA-1.5. Count the budget violations.
Feasibility. Sweep $\lambda$ densely and plot every slice. Then ask whether any $\lambda$ satisfies all budgets that the multiplier method satisfies.
Composite triggers. Compare the joint adversary against the image-only adversary and against plain fine-tuning.
If the constrained method loses on transfer, the core claim is wrong. If it wins only on composite triggers, the contribution is the joint adversary rather than the budgets, and the note would be reframed around that.
References
- VL-Trojan: Multimodal Instruction Backdoor Attacks against Autoregressive Visual Language Models
- Fine-Pruning: Defending Against Backdooring Attacks on Deep Neural Networks
- Adversarial Neuron Pruning Purifies Backdoored Deep Models
- Unlearning Backdoor Threats: Enhancing Backdoor Defense in Multimodal Contrastive Learning via Local Token Unlearning
- Adversarial Backdoor Defense in CLIP
- Obliviate: Neutralizing Task-agnostic Backdoors within the Parameter-efficient Fine-tuning Paradigm
- Fast and Stable Interval Bounds Propagation for Training Verifiably Robust Models
- Expressive Losses for Verified Robustness via Convex Combinations
- USB: A Comprehensive and Unified Safety Evaluation Benchmark for Multimodal Large Language Models
- Safe Inputs but Unsafe Output: Benchmarking Cross-modality Safety Alignment of Large Vision-Language Models
- Adversarial Robustness with Semi-Infinite Constrained Learning
- Adversarial Unlearning of Backdoors via Implicit Hypergradient
- Vision-and-Language or Vision-for-Language? On Cross-Modal Influence in Multimodal Transformers
- BackdoorMBTI: A Backdoor Learning Multimodal Benchmark Tool Kit for Backdoor Defense Evaluation
- Images are Achilles' Heel of Alignment: Exploiting Visual Vulnerabilities for Jailbreaking Multimodal Large Language Models
- Architectural Backdoors in Vision-Language Model Supply Chains via Representation Steering
- Baseline Defenses for Adversarial Attacks Against Aligned Language Models