Bootstrapping LLM Robustness for VLM Safety via Reducing the Pretraining Modality Gap

Wenhan Yang, Spencer Stice, Ali Payani, Baharan MirzasoleimanarXiv 2025 · DOI · arXiv

Abstract

Large Vision-Language Models (LVLMs) often suffer from "safety degradation," where a robustly aligned text-only Large Language Model (LLM) becomes vulnerable to harmful prompts when visual inputs are introduced. This research identifies the "modality gap"—the representational distance between image and text embeddings—as a primary driver of this vulnerability. We introduce REGAP (REgularizing the modality GAP), a simple regularization technique applied during the pretraining phase to pull these representations closer together. Our results demonstrate that reducing this gap significantly enhances model robustness against jailbreaks across multiple architectures without compromising general utility or requiring specialized safety datasets.

The Problem: Safety Degradation in VLMs

Large Language Models like Llama or Vicuna undergo rigorous safety alignment to prevent the generation of harmful content. However, when these models are integrated with vision encoders to create LVLMs, that safety often evaporates. Surprisingly, even a blank or irrelevant image can act as a "trigger," causing a VLM to provide a harmful response to a prompt it would have correctly refused in a text-only setting.

While researchers previously suspected that the "modality gap"—the fact that image and text embeddings occupy different neighborhoods in the model's internal space—played a role, this work is the first to rigorously quantify that relationship and provide a training-time solution.

Methods: Regularizing the Modality Gap (REGAP)

The authors propose REGAP to address safety at its structural root. The methodology consists of four key components:

  1. Quantifying the Gap: The researchers utilize the Modality Integration Rate (MIR), which employs Fréchet Inception Distance (FID) to measure how well image and text distributions overlap across the model's layers.
  2. The REGAP Regularizer: During the pretraining phase—where the model learns to map visual features to the LLM's input space—a new regularization term is added to the loss function. This term calculates the pairwise $L_2$ distance between all image tokens and all text tokens in a given pair to force them into a shared representational space.
  3. Dynamic Scaling: To prevent the regularization loss from drowning out standard cross-entropy learning, a scaling factor ($\alpha$) is calculated during a "warm-up" phase to ensure the losses are of comparable magnitude.
  4. Layer Targeting: Crucially, the study found that reducing the gap at the input layer (the projector) is the most effective. Attempting to force alignment in deeper layers or during final instruction fine-tuning actually degraded performance.
Figure 5: Visualization of the modality gap, showing the significant distance between Image-Prompt and Prompt-Prompt pairwise L2 distances in standard models.

Experimental Setup

The study utilized a robust suite of benchmarks to test both safety and utility:

  • Safety Benchmarks: HADES (featuring Toxic and Adversarial images), MM-SafetyBench (typographic harmful intent), and FigStep.
  • Utility Benchmarks: GQA, MMBench, ScienceQA, SEEDBench, and TextVQA to ensure no loss in general performance.
  • Architectures: The method was validated on LLaVA v1.5, ShareGPT4V, and MiniGPT-4.

Key Insights and Results

The Modality Gap-Safety Correlation

A central finding of this research is the high correlation between the modality gap and safety failures. As shown in the figures below, there is a strong relationship (up to R=0.78) between the Modality Integration Rate (MIR) and the unsafe rate of the model. Essentially, the larger the gap, the easier the model is to jailbreak.

Figure 1: Correlations between MIR and Unsafe Rates. (a) and (b) show that higher MIR (smaller gap) correlates with lower unsafe rates, while (c) shows the gap persists from pretraining (PT) to fine-tuning (FT).

Performance Gains

  • Significant Safety Improvements: On LLaVA-v1.5-7B-LoRA, REGAP reduced the unsafe rate by 16.3% on the HADES Toxic dataset. MiniGPT-4 saw an even larger reduction of 24.3%.
  • Utility Preservation: Across 10 utility benchmarks, REGAP-trained models performed within 1% of baseline models, with some slight improvements in benchmarks like MMStar due to tighter alignment.
  • Efficiency: REGAP adds negligible computational overhead, increasing training time by only roughly 1.05x.

Conclusion and Future Work

REGAP demonstrates that VLM safety is not just a matter of fine-tuning with better data, but is deeply tied to the fundamental architecture of the representation space. By closing the modality gap during pretraining, we can bootstrap the existing robustness of LLMs for the vision-language domain.

Future research will investigate why regularizing deeper layers of the LLM harms performance, as well as how these dynamics scale to massive models (70B+) that may naturally exhibit different modality gap characteristics.

Summary updated

Abstract

Ensuring Vision-Language Models (VLMs) generate safe outputs is crucial for their reliable deployment. However, LVLMs suffer from drastic safety degradation compared to their LLM backbone. Even blank or irrelevant images can trigger LVLMs to generate harmful responses to prompts that would otherwise be refused in text-only contexts. The modality gap between image and text representations has been recently hypothesized to contribute to safety degradation of LVLMs. However, if and how the amount of modality gap affects LVLMs' safety is not studied. In this work, we show that the amount of modality gap is highly inversely correlated with VLMs' safety. Then, we show that this modality gap is introduced during pretraining LVLMs and persists through fine-tuning. Inspired by this observation, we propose a regularization to reduce the modality gap during pretraining. Our extensive experiments on LLaVA v1.5, ShareGPT4V, and MiniGPT-4 show that our method substantially improves safety alignment of LVLMs, reducing unsafe rate by up to 16.3% without compromising performance, and can further boost existing defenses by up to 18.2%.
BibTeX
@misc{yang2025bootstrapping,
  author = {Wenhan Yang and Spencer Stice and Ali Payani and Baharan Mirzasoleiman},
  title = {{Bootstrapping LLM Robustness for VLM Safety via Reducing the Pretraining Modality Gap}},
  year = {2025},
  eprint = {2505.24208},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2505.24208},
}
Figure