TL;DR
Instead of guessing reward weights to make Vision-Language Models (VLMs) safe, this paper proposes setting explicit "safety budgets" for harm and over-refusal rates. By introducing a two-channel architecture for Constrained Group Relative Policy Optimization (GRPO), the authors fix systematic scorer biases and ensure that safety constraints actually influence the model's learning process.
What Problem Does This Paper Address?
Safety tuning for VLMs is often a "black art." Researchers typically mix a safety reward with a helpfulness reward using a manually tuned weight. If the model is too dangerous, you increase the weight; if it starts refusing harmless questions (over-refusal), you decrease it. This trial-and-error process is inefficient and provides no guarantee on the final safety level.
The principled solution is constrained optimization: tell the model to "be as helpful as possible, provided the harm rate is below 5%." This is usually done using Lagrange multipliers that automatically adjust the safety penalty during training. However, the authors discovered that this "natural fix" fails twice when applied to the popular GRPO framework:
- The Scorer Bias Problem: Most safety scorers (like LLM-as-a-judge) are trained to rank responses, not to provide absolute probability estimates. If a scorer consistently overestimates harm by 10%, a 5% budget becomes impossible to meet, causing the safety penalty to skyrocket and ruin the model.
- The Multiplier Cancellation Problem: Standard GRPO standardizes rewards within a group of responses. Mathematically, this standardization often cancels out the Lagrange multiplier before it can affect the gradient, effectively "muting" the safety signal.
Methods
The authors propose Two-Channel Constrained GRPO to repair these failures. The architecture separates how the scorer is used for ranking versus how it's used for control.
- Calibration (The Control Channel): To fix scorer bias, they fit a simple monotone map (using isotonic regression) on a small set of ~200 judge-labeled responses. This map translates the scorer's raw output into a calibrated "rate" that the budget can actually understand.
- Ranking (The Learning Channel): While the budget needs a calibrated rate, the policy gradient needs fine-grained ranking. The authors keep the raw, uncalibrated scores in the base reward to preserve the model's ability to distinguish between "slightly bad" and "very bad" responses.
- Advantage Scaling: To prevent the multiplier from being canceled out, they use "unscaled advantages" (setting the group standard deviation to 1) or standardize each objective (helpfulness vs. safety) separately before weighting them. This ensures the Lagrange multiplier $\lambda$ actually scales the gradient updates.
Datasets
- Training: The model (Qwen3-VL-2B) was trained on 130 "hard" harmful prompts from MM-SafetyBench and 200 benign queries from MOSSBench.
- Evaluation: A massive "held-out" set of 1,037 adversarial prompts across four benchmarks: MM-SafetyBench, FigStep, JailbreakV-28K, and the MOSSBench contrast set.
- Generalization: They tested on the HADES benchmark, which uses optimized adversarial images that were never seen during training or calibration.
- Capabilities: MMBench was used to ensure that safety tuning didn't degrade the model's general vision-language reasoning.
Key Insights and Ideas
- The "Three States" of a Controller: The authors prove (Proposition 1) that a scorer's mean bias, not the budget itself, determines the controller's fate. A biased scorer will either drive the penalty to its maximum cap (saturated), switch it off entirely (inactive), or—if calibrated—allow it to settle at a useful interior point.
- Calibration is a "Control Knob": Once calibrated, the budget acts as a literal dial. Tightening the harm budget from 0.20 to 0.05 predictably and monotonically reduces the measured harm on held-out data.
- Text-Only Scorers for Visual Attacks: Interestingly, the safety scorer only needs to read the model's text response. Even when the harmful intent is hidden in an image, a model that complies will usually "name" the harmful activity in its response, allowing the text-based scorer to catch it.
Key Results
The two-channel approach significantly outperformed standard GRPO and uncalibrated constrained methods:
- Harm Reduction: On 1,037 adversarial prompts, held-out harm fell from 8.9% (standard GRPO) to 4.8%.
- Image-Borne Attacks: The gains were most dramatic where intent was hidden in images (e.g., a "fake ID" card), with harm dropping from 10.1% to 5.1%.
- Unseen Robustness: On the difficult HADES benchmark, harm was slashed from 28.9% to 12.8%.
- Minimal Side Effects: Over-refusal of benign queries remained at a low 2%, and general performance on MMBench was virtually unchanged.
Limitations and Future Work
- Label Requirement: The method requires a few hundred human or high-quality LLM-judge labels to fit the calibration map. While small, this is still a manual step.
- Static Calibration: The authors fit the map once on an unconstrained run. While they show this holds up well, a more dynamic "online" calibration that updates during training could potentially be even more robust.
- Scorer Modality: While text-only scoring worked well here, extremely subtle visual attacks that don't trigger explicit text in the response might still require a true multimodal safety scorer.
Summary updated