Introduction
Safety alignment is a critical component in the development of Large Language Models (LLMs), ensuring they refuse harmful requests. However, Vision-Language Models (VLMs) face a unique challenge: adversarial visual inputs. Even when the text-based safeguards are robust, an attacker can inject imperceptible "adversarial noise" into an image to bypass these protections. This paper introduces Adversary-Aware DPO (ADPO), a novel framework designed to close this security gap by integrating adversarial training directly into the Direct Preference Optimization (DPO) pipeline.
The ADPO Framework
The ADPO framework hardens VLMs against white-box jailbreak attacks through two primary innovations: a robust reference model and a modified loss function.
Adversarial-trained Reference Model (AR-DPO)
In standard DPO, a frozen reference model provides the baseline for optimization. If this reference model is vulnerable to adversarial noise, the training signal becomes unreliable. ADPO first constructs an Adversarial-trained Reference Model (AR-DPO) using a min-max optimization strategy. This process identifies worst-case image or latent space perturbations and trains the model to maintain safe, human-preferred responses despite these distortions.
Adversarial-aware DPO Loss (AT-DPO)
The training objective is further refined through the Adversarial-aware DPO Loss (AT-DPO). Unlike standard DPO, which compares responses on clean data, AT-DPO explicitly incorporates adversarial perturbations into the loss calculation. This encourages the target model to assign high probability to safe answers even when the input is being actively manipulated by an adversary.
Experimental Evaluation
The effectiveness of ADPO was tested on LLaVA-1.5 and LLaVA-1.6 architectures using a combination of safety and utility datasets.
- Safety Training: 200 harmful image-text pairs from HarmBench-mm and HarmBench-AT.
- Utility Training: 500 samples from LLaVA-Instruct-150K to prevent over-optimization for safety.
- Benchmarks: Safety was evaluated using VisualAdv, MMPGDBlank, and MultiTrust. Utility was measured via MMStar, OCRBench, MM-Vet, and LLaVABench.
Results
ADPO demonstrates a significant reduction in the success rate of adversarial attacks while maintaining high general utility.
Safety Robustness
On the challenging MMPGDBlank attack, standard DPO exhibited a 33% success rate on LLaVA-1.5. ADPO reduced this to a negligible 0.5%. On LLaVA-1.6, the attack success rate dropped from 7.0% to 0.0%. ADPO also consistently outperformed baselines across the MultiTrust benchmark, proving resilient against typographic and crossmodal jailbreaks.
Utility Preservation
A common challenge in safety alignment is the "safety tax," where increased robustness leads to decreased helpfulness. As shown in the radar chart, ADPO maintains competitive scores across utility benchmarks like OCRBench and MM-Vet. In some instances, such as the MMStar benchmark, ADPO even showed slight performance gains, suggesting that adversarial training can enhance the model's overall representational robustness.
Conclusion
Adversary-Aware DPO (ADPO) represents a significant step forward in securing Vision-Language Models. By hardening the reference model and incorporating adversarial awareness into the preference optimization loss, ADPO effectively neutralizes sophisticated white-box attacks with minimal impact on model utility. This framework provides a scalable and effective method for building safer, more resilient multimodal AI systems.
Summary updated