Introduction
Large Vision-Language Models (LVLMs) like Flamingo and LLaVA have revolutionized multimodal tasks. However, their reliance on instruction tuning makes them highly vulnerable to backdoor attacks. In these attacks, a malicious actor poisons the training dataset with hidden visual or textual triggers that force the model to generate harmful or specific incorrect outputs.
Existing backdoor defenses often fail in the context of LVLMs because they typically require updating the entire model's parameters. In practice, LVLMs are fine-tuned using parameter-efficient methods where the massive vision encoder and language model backbones remain frozen, and only small "adapter" modules are updated. This paper introduces RobustIT, a lightweight defense designed specifically for this adapter-based tuning paradigm.
The RobustIT Framework
RobustIT is an attack-agnostic framework that operates exclusively on the trainable adapter modules and text-embedding layers. It employs two primary regularization techniques to neutralize backdoor triggers without requiring prior knowledge of the attack type.
- Input Diversity Regularization (IDR): Backdoor attacks typically rely on the model memorizing a precise, brittle mapping between a fixed trigger and a malicious response. IDR disrupts this by applying random augmentations—such as color jittering and flipping for images, or synonym replacement for text—during training. An intra-modal consistency loss then forces the model to maintain stable representations despite these perturbations, preventing the model from overfitting to specific trigger patterns.
- Anomalous Activation Regularization (AAR): The authors discovered that backdoor triggers cause abnormally high activation spikes in specific adapter channels. AAR monitors these spikes using a momentum-based importance score and applies a sparse binary mask to suppress over-active channels. This effectively starves the backdoor of the gradient flow necessary to embed itself in the model weights.
Experimental Evaluation
The researchers evaluated RobustIT using the MIMIC-IT dataset (2.8 million triplets) for training and standard benchmarks like MSCOCO and Flickr30k for evaluation. They tested the framework against seven diverse backdoor attacks: BadNets, Blended, SIG, SSBA, FTrojan, TrojVQA, and VLTrojan, with a poisoning rate of 1%.
Key Results
RobustIT demonstrated exceptional performance in both security and utility. Across all tested attacks, the framework reduced the Attack Success Rate (ASR) to near-zero levels. For instance, against the VLTrojan attack on MSCOCO, the ASR dropped from a staggering 99.1% to just 0.44%.
Crucially, this security comes with no performance penalty. In many cases, RobustIT actually improved standard performance metrics by encouraging the model to learn more robust semantic representations. On clean MSCOCO data, the CIDEr score improved from 48.0 to 55.3. Furthermore, the framework is highly efficient, adding less than 15% computational overhead to the training process.
Conclusion
RobustIT provides a practical and effective solution for securing LVLMs during instruction tuning. By focusing on adapter-level regularizations, it blocks complex multimodal backdoors while preserving—and even enhancing—the model's primary task performance. Future work will explore the limits of activation sparsity and the potential for this framework to improve general model alignment.
Summary updated