Robust Anti-Backdoor Instruction Tuning in LVLMs

Yuan Xun, Siyuan Liang, Xiaojun Jia, Xinwei Liu, Xiaochun CaoarXiv 2025 · DOI · arXiv

Introduction

Large Vision-Language Models (LVLMs) like Flamingo and LLaVA have revolutionized multimodal tasks. However, their reliance on instruction tuning makes them highly vulnerable to backdoor attacks. In these attacks, a malicious actor poisons the training dataset with hidden visual or textual triggers that force the model to generate harmful or specific incorrect outputs.

Existing backdoor defenses often fail in the context of LVLMs because they typically require updating the entire model's parameters. In practice, LVLMs are fine-tuned using parameter-efficient methods where the massive vision encoder and language model backbones remain frozen, and only small "adapter" modules are updated. This paper introduces RobustIT, a lightweight defense designed specifically for this adapter-based tuning paradigm.

The RobustIT Framework

RobustIT is an attack-agnostic framework that operates exclusively on the trainable adapter modules and text-embedding layers. It employs two primary regularization techniques to neutralize backdoor triggers without requiring prior knowledge of the attack type.

The RobustIT framework architecture, illustrating Input Diversity Regularization (IDR) and Anomalous Activation Regularization (AAR).
  1. Input Diversity Regularization (IDR): Backdoor attacks typically rely on the model memorizing a precise, brittle mapping between a fixed trigger and a malicious response. IDR disrupts this by applying random augmentations—such as color jittering and flipping for images, or synonym replacement for text—during training. An intra-modal consistency loss then forces the model to maintain stable representations despite these perturbations, preventing the model from overfitting to specific trigger patterns.
  2. Anomalous Activation Regularization (AAR): The authors discovered that backdoor triggers cause abnormally high activation spikes in specific adapter channels. AAR monitors these spikes using a momentum-based importance score and applies a sparse binary mask to suppress over-active channels. This effectively starves the backdoor of the gradient flow necessary to embed itself in the model weights.

Experimental Evaluation

The researchers evaluated RobustIT using the MIMIC-IT dataset (2.8 million triplets) for training and standard benchmarks like MSCOCO and Flickr30k for evaluation. They tested the framework against seven diverse backdoor attacks: BadNets, Blended, SIG, SSBA, FTrojan, TrojVQA, and VLTrojan, with a poisoning rate of 1%.

Key Results

RobustIT demonstrated exceptional performance in both security and utility. Across all tested attacks, the framework reduced the Attack Success Rate (ASR) to near-zero levels. For instance, against the VLTrojan attack on MSCOCO, the ASR dropped from a staggering 99.1% to just 0.44%.

Radar charts comparing Vanilla Instruction Tuning (VanillaIT) and RobustIT across multiple metrics and attack types. RobustIT consistently maintains a larger area, indicating superior performance and robustness.

Crucially, this security comes with no performance penalty. In many cases, RobustIT actually improved standard performance metrics by encouraging the model to learn more robust semantic representations. On clean MSCOCO data, the CIDEr score improved from 48.0 to 55.3. Furthermore, the framework is highly efficient, adding less than 15% computational overhead to the training process.

Conclusion

RobustIT provides a practical and effective solution for securing LVLMs during instruction tuning. By focusing on adapter-level regularizations, it blocks complex multimodal backdoors while preserving—and even enhancing—the model's primary task performance. Future work will explore the limits of activation sparsity and the potential for this framework to improve general model alignment.

Summary updated

Abstract

Large visual language models (LVLMs) have demonstrated excellent instruction-following capabilities, yet remain vulnerable to stealthy backdoor attacks when finetuned using contaminated data. Existing backdoor defense techniques are usually developed for single-modal visual or language models under fully parameter-adjustable settings or rely on supervisory knowledge during training. However, in real-world scenarios, defenders cannot modify frozen visual encoders or core LLM parameters, nor possess prior knowledge of unknown trigger patterns or target responses. Motivated by the empirical finding that LVLMs readily overfit to fixed, unknown triggers, which can embed malicious associations during adapter-level tuning, we aim to design a defense that operates without access to core weights or attack priors. To this end, we introduce a lightweight, certified-agnostic defense framework, Robust Instruction Tuning, that finetunes only adapter modules and text embedding layers under instruction tuning. Our method integrates two complementary regularizations: (1) Input Diversity Regularization, which perturbs trigger components across training samples to disrupt consistent spurious cues; and (2) Anomalous Activation Regularization, which dynamically sparses adapter weights exhibiting abnormally sharp activations linked to backdoor patterns. These mechanisms jointly guide the model toward learning semantically grounded representations rather than memorizing superficial trigger-response mappings. Extensive experiments against seven attacks on Flickr30k and MSCOCO demonstrate that ours reduces their attack success rate to nearly zero, with an increase in training cost of less than 15%.
BibTeX
@misc{xun2025robust,
  author = {Yuan Xun and Siyuan Liang and Xiaojun Jia and Xinwei Liu and Xiaochun Cao},
  title = {{Robust Anti-Backdoor Instruction Tuning in LVLMs}},
  year = {2025},
  eprint = {2506.05401},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2506.05401},
}
Figure