Safety as a Budget: Calibrating the Control Channel of Constrained GRPO for Vision-Language Models

Bo HanarXiv 2026 · arXiv

TL;DR

Instead of guessing reward weights to make Vision-Language Models (VLMs) safe, this paper proposes setting explicit "safety budgets" for harm and over-refusal rates. By introducing a two-channel architecture for Constrained Group Relative Policy Optimization (GRPO), the authors fix systematic scorer biases and ensure that safety constraints actually influence the model's learning process.

What Problem Does This Paper Address?

Safety tuning for VLMs is often a "black art." Researchers typically mix a safety reward with a helpfulness reward using a manually tuned weight. If the model is too dangerous, you increase the weight; if it starts refusing harmless questions (over-refusal), you decrease it. This trial-and-error process is inefficient and provides no guarantee on the final safety level.

The principled solution is constrained optimization: tell the model to "be as helpful as possible, provided the harm rate is below 5%." This is usually done using Lagrange multipliers that automatically adjust the safety penalty during training. However, the authors discovered that this "natural fix" fails twice when applied to the popular GRPO framework:

  1. The Scorer Bias Problem: Most safety scorers (like LLM-as-a-judge) are trained to rank responses, not to provide absolute probability estimates. If a scorer consistently overestimates harm by 10%, a 5% budget becomes impossible to meet, causing the safety penalty to skyrocket and ruin the model.
  2. The Multiplier Cancellation Problem: Standard GRPO standardizes rewards within a group of responses. Mathematically, this standardization often cancels out the Lagrange multiplier before it can affect the gradient, effectively "muting" the safety signal.

Methods

The authors propose Two-Channel Constrained GRPO to repair these failures. The architecture separates how the scorer is used for ranking versus how it's used for control.

Figure 2. Two-channel constrained GRPO. (a) A scorer reads the request and the response. (b) Raw probabilities set the base reward r0; the penalty uses the mapped level c(p), and unscaled advantages let the multipliers scale the gradient. (c) The map sets the level the dual update compares with the budgets.
  • Calibration (The Control Channel): To fix scorer bias, they fit a simple monotone map (using isotonic regression) on a small set of ~200 judge-labeled responses. This map translates the scorer's raw output into a calibrated "rate" that the budget can actually understand.
  • Ranking (The Learning Channel): While the budget needs a calibrated rate, the policy gradient needs fine-grained ranking. The authors keep the raw, uncalibrated scores in the base reward to preserve the model's ability to distinguish between "slightly bad" and "very bad" responses.
  • Advantage Scaling: To prevent the multiplier from being canceled out, they use "unscaled advantages" (setting the group standard deviation to 1) or standardize each objective (helpfulness vs. safety) separately before weighting them. This ensures the Lagrange multiplier $\lambda$ actually scales the gradient updates.

Datasets

  • Training: The model (Qwen3-VL-2B) was trained on 130 "hard" harmful prompts from MM-SafetyBench and 200 benign queries from MOSSBench.
  • Evaluation: A massive "held-out" set of 1,037 adversarial prompts across four benchmarks: MM-SafetyBench, FigStep, JailbreakV-28K, and the MOSSBench contrast set.
  • Generalization: They tested on the HADES benchmark, which uses optimized adversarial images that were never seen during training or calibration.
  • Capabilities: MMBench was used to ensure that safety tuning didn't degrade the model's general vision-language reasoning.

Key Insights and Ideas

  • The "Three States" of a Controller: The authors prove (Proposition 1) that a scorer's mean bias, not the budget itself, determines the controller's fate. A biased scorer will either drive the penalty to its maximum cap (saturated), switch it off entirely (inactive), or—if calibrated—allow it to settle at a useful interior point.
Visualizing the three states: overestimating bias saturates the multiplier, underestimating switches it off, and calibration lets it settle.
  • Calibration is a "Control Knob": Once calibrated, the budget acts as a literal dial. Tightening the harm budget from 0.20 to 0.05 predictably and monotonically reduces the measured harm on held-out data.
  • Text-Only Scorers for Visual Attacks: Interestingly, the safety scorer only needs to read the model's text response. Even when the harmful intent is hidden in an image, a model that complies will usually "name" the harmful activity in its response, allowing the text-based scorer to catch it.

Key Results

The two-channel approach significantly outperformed standard GRPO and uncalibrated constrained methods:

  • Harm Reduction: On 1,037 adversarial prompts, held-out harm fell from 8.9% (standard GRPO) to 4.8%.
  • Image-Borne Attacks: The gains were most dramatic where intent was hidden in images (e.g., a "fake ID" card), with harm dropping from 10.1% to 5.1%.
  • Unseen Robustness: On the difficult HADES benchmark, harm was slashed from 28.9% to 12.8%.
  • Minimal Side Effects: Over-refusal of benign queries remained at a low 2%, and general performance on MMBench was virtually unchanged.
Comparison of held-out harm across different intent locations and the unseen HADES benchmark.

Limitations and Future Work

  • Label Requirement: The method requires a few hundred human or high-quality LLM-judge labels to fit the calibration map. While small, this is still a manual step.
  • Static Calibration: The authors fit the map once on an unconstrained run. While they show this holds up well, a more dynamic "online" calibration that updates during training could potentially be even more robust.
  • Scorer Modality: While text-only scoring worked well here, extremely subtle visual attacks that don't trigger explicit text in the response might still require a true multimodal safety scorer.

Summary updated

Abstract

Reinforcement learning can teach vision-language models (VLMs) to decline harmful image-text requests, but how safe the result becomes is set indirectly, through reward weights found by trial runs. Stating safety as budgets on the rate of harmful assistance and the rate of over-refusal, enforced by Lagrange multipliers inside GRPO with an off-the-shelf scorer as the cost, is the natural fix, and it fails twice. The multiplier compares the scorer's mean with the budget, and scorers built for ranking get the mean wrong: a fixed-point analysis shows that the mean bias, not the budget, then decides whether the multiplier settles, saturates or switches off. And standard GRPO standardizes the penalized reward within each group, which cancels the multiplier before it reaches the gradient. Two-channel constrained GRPO repairs both: the base reward keeps the raw score for the within-group comparison, the penalty and the constraint estimate pass through a monotone map fitted on judge-labeled training responses, and the advantage keeps the multiplier. On a Qwen3-VL-2B policy with a training-free readout of an open 4B model as the scorer, held-out harm on 1,037 adversarial prompts falls from 8.9% under standard GRPO to 4.8%, lower in every seed, where the uncalibrated budget gives 7.6%; over-refusal stays at most 2% on 400 benign queries and MMBench is unchanged. The gain is largest where the intent sits in the image (10.1% -> 5.1%) and grows on the unseen HADES benchmark (28.9% -> 12.8%). Order-preserving distortions of the scores reproduce each predicted controller state in every seed, and the results hold under a map fitted on another judge's labels, an image-reading judge and a human audit.
BibTeX
@misc{han2026safety,
  author = {Bo Han},
  title = {{Safety as a Budget: Calibrating the Control Channel of Constrained GRPO for Vision-Language Models}},
  year = {2026},
  eprint = {2611.16628},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.16628},
}
Figure