Abstract
Large Vision-Language Models (LVLMs) often suffer from "safety degradation," where a robustly aligned text-only Large Language Model (LLM) becomes vulnerable to harmful prompts when visual inputs are introduced. This research identifies the "modality gap"—the representational distance between image and text embeddings—as a primary driver of this vulnerability. We introduce REGAP (REgularizing the modality GAP), a simple regularization technique applied during the pretraining phase to pull these representations closer together. Our results demonstrate that reducing this gap significantly enhances model robustness against jailbreaks across multiple architectures without compromising general utility or requiring specialized safety datasets.
The Problem: Safety Degradation in VLMs
Large Language Models like Llama or Vicuna undergo rigorous safety alignment to prevent the generation of harmful content. However, when these models are integrated with vision encoders to create LVLMs, that safety often evaporates. Surprisingly, even a blank or irrelevant image can act as a "trigger," causing a VLM to provide a harmful response to a prompt it would have correctly refused in a text-only setting.
While researchers previously suspected that the "modality gap"—the fact that image and text embeddings occupy different neighborhoods in the model's internal space—played a role, this work is the first to rigorously quantify that relationship and provide a training-time solution.
Methods: Regularizing the Modality Gap (REGAP)
The authors propose REGAP to address safety at its structural root. The methodology consists of four key components:
- Quantifying the Gap: The researchers utilize the Modality Integration Rate (MIR), which employs Fréchet Inception Distance (FID) to measure how well image and text distributions overlap across the model's layers.
- The REGAP Regularizer: During the pretraining phase—where the model learns to map visual features to the LLM's input space—a new regularization term is added to the loss function. This term calculates the pairwise $L_2$ distance between all image tokens and all text tokens in a given pair to force them into a shared representational space.
- Dynamic Scaling: To prevent the regularization loss from drowning out standard cross-entropy learning, a scaling factor ($\alpha$) is calculated during a "warm-up" phase to ensure the losses are of comparable magnitude.
- Layer Targeting: Crucially, the study found that reducing the gap at the input layer (the projector) is the most effective. Attempting to force alignment in deeper layers or during final instruction fine-tuning actually degraded performance.
Experimental Setup
The study utilized a robust suite of benchmarks to test both safety and utility:
- Safety Benchmarks: HADES (featuring Toxic and Adversarial images), MM-SafetyBench (typographic harmful intent), and FigStep.
- Utility Benchmarks: GQA, MMBench, ScienceQA, SEEDBench, and TextVQA to ensure no loss in general performance.
- Architectures: The method was validated on LLaVA v1.5, ShareGPT4V, and MiniGPT-4.
Key Insights and Results
The Modality Gap-Safety Correlation
A central finding of this research is the high correlation between the modality gap and safety failures. As shown in the figures below, there is a strong relationship (up to R=0.78) between the Modality Integration Rate (MIR) and the unsafe rate of the model. Essentially, the larger the gap, the easier the model is to jailbreak.
Performance Gains
- Significant Safety Improvements: On LLaVA-v1.5-7B-LoRA, REGAP reduced the unsafe rate by 16.3% on the HADES Toxic dataset. MiniGPT-4 saw an even larger reduction of 24.3%.
- Utility Preservation: Across 10 utility benchmarks, REGAP-trained models performed within 1% of baseline models, with some slight improvements in benchmarks like MMStar due to tighter alignment.
- Efficiency: REGAP adds negligible computational overhead, increasing training time by only roughly 1.05x.
Conclusion and Future Work
REGAP demonstrates that VLM safety is not just a matter of fine-tuning with better data, but is deeply tied to the fundamental architecture of the representation space. By closing the modality gap during pretraining, we can bootstrap the existing robustness of LLMs for the vision-language domain.
Future research will investigate why regularizing deeper layers of the LLM harms performance, as well as how these dynamics scale to massive models (70B+) that may naturally exhibit different modality gap characteristics.
Summary updated