Vision-language models learn to refuse harmful requests mainly by being trained to emit refusal text such as "I cannot help with that." In effect, the decision "should I answer?" is learned as ordinary next-token prediction with cross-entropy. That loss is convex. Theory for adversarial classification shows that no convex surrogate loss is adversarially consistent: minimizing it does not minimize the true error under attack. The reason is that convex losses reward "fence-sitting," meaning low-confidence predictions sitting right at the decision boundary. We propose to take the refuse-or-comply decision out of the text stream and into an explicit decision variable with three outcomes: comply, refuse, and abstain. We train it with a non-convex loss that the reject-option literature has shown to be calibrated under adversarial perturbation.
The bet is concrete. Much of today's over-refusal of benign-but-borderline multimodal inputs, and much of the brittleness to attack styles not seen in training, may come from the training loss rather than from any unavoidable safety–utility trade-off. If so, a calibrated abstention gate should refuse fewer harmless prompts than a cross-entropy gate at the same attack success rate. It should also hold up better against attack families it never saw in training. If it does not, that is a useful negative result: it would mean surrogate consistency theory does not transfer to foundation-model safety.
Core Claim
In multi-class vision, Theoretically Grounded Loss Functions and Algorithms for Adversarial Robustness proved that TRADES is not consistent even for linear models. It showed that a consistent smooth surrogate (PSAL) raises clean and robust accuracy together. On CIFAR-10 at WRN-70-16 this was 86.63% clean versus 85.34% for TRADES, at 57.46% versus 57.05% AutoAttack accuracy. This paper motivates our work: it is evidence that part of the "clean accuracy tax" is an artifact of the loss.
Towards Calibrated Losses for Adversarial Robust Reject Option Classification supplies the method. For the adversarial reject-option loss with rejection cost $d$ and attack radius $\gamma$, it shows convex surrogates fail because they lack a "minima-jump." That is an abrupt switch from abstaining to a confident output. Its Shifted Double Sigmoid and Shifted Double Ramp losses restore the switch, but only for linear classifiers on 2D synthetic data.
Our claim joins these two lines. A refusal gate on top of a multimodal model, trained with a shifted, adversarially calibrated reject-option loss, will beat a cross-entropy-trained gate of identical architecture on two measures at matched vulnerability: benign false refusal, and held-out attack success. The gain should show up specifically as fewer benign inputs sitting near the gate's decision boundary.
Concretely, let $g(x)$ be a scalar gate score computed from the model's hidden state for image–text input $x$, and let $y \in \{-1,+1\}$ mark harmful versus benign. The gate complies when $g(x) > \rho$, refuses when $g(x) < -\rho$, and abstains otherwise. Abstention routes to a conservative completion mode and costs $d$. The target risk is
$$ R_{d,\gamma}(g) = \mathbb{E}\Big[\max_{\|\delta\|\le\gamma} \ell_d\big(y, g(x+\delta)\big)\Big], $$
where $\ell_d$ is 1 for a wrong comply/refuse decision, $d$ for abstention, and 0 otherwise. The perturbation $\delta$ is taken over image pixels or input embeddings. The proposal is to minimize a shifted ramp or sigmoid surrogate of $R_{d,\gamma}$ rather than a cross-entropy surrogate.
Relation to Prior Work
Connection D in the literature already attacks the multimodal safety–utility tension, but empirically.
- USB benchmarks 17 models across image/text risk combinations. It finds that the safest model, Claude 3.5 Sonnet (15.57% attack success rate), is also the most oversensitive (25.82% refusal of harmless prompts). USB motivates our work but proposes no training method.
- Towards Robust Multimodal Large Language Models Against Jailbreak Attacks (SafeMLLM) does adversarial training with a contrastive embedding attack and LoRA, regularized by a benign-utility loss. It is our strongest training-time competitor. Its refusal remains a token-level contrastive objective with no consistency analysis, which is exactly the object we replace.
- Reimagining Safety Alignment with An Image (Magic Image) shifts the refusal boundary with an optimized visual prompt. It shows geometry alone can reduce over-refusal, but it gives no guarantees and can be overridden by an adversary's own image.
The generalization side comes from Confidence-Calibrated Adversarial Training (CCAT). CCAT trains vision classifiers to decay toward uniform confidence away from clean data and then reject by threshold. On SVHN this cut unseen-$\ell_2$ error from 99.4% (standard adversarial training) to 29.0%. We borrow its insight that a trained abstention region generalizes across threat models. We differ in two ways: we train that region with a calibrated surrogate rather than a confidence schedule, and we apply it to a generative model's safety decision.
Adversarial Robustness in One-Stage Learning-to-Defer constrains our design. It shows that jointly trained deferral routing is itself an attack surface, so our evaluation must attack the gate and the generator together.
Finally, the impossibility results in The Adversarial Consistency of Surrogate Risks for Binary Classification and Towards Consistency in Adversarial Classification are what make the cross-entropy gate a principled baseline rather than a straw man. They predict failure at the boundary specifically. However, all of these proofs are for linear or low-dimensional settings, and we do not claim they carry over to deep networks. Testing whether they matter there is the point of the work.
Research Program
We build the gate as a small head on the pooled hidden state at the final prompt token of LLaVA-1.5-13B, the model SafeMLLM reports on. We train two variants: one with the base model frozen, and one with LoRA on the projector and upper layers. The gate's decision is enforced at generation time. A "refuse" decision emits a fixed refusal. An "abstain" decision prepends a cautious system instruction and caps output length. A "comply" decision generates normally.
Training data mixes harmful and benign image–text pairs from the USB-Base split (USB resource), deliberately including benign-borderline items. The inner maximization uses $\ell_\infty$ pixel perturbations and the embedding-space attack of SafeMLLM. Typographic, safe-image/safe-text compositional, and text-suffix attacks are never seen in training.
The program has three stages.
- Synthetic replication. We first replicate the minima-jump phenomenon on frozen gate features, where the geometry is close to linear. This tells us whether the shifted losses behave as theory predicts before we pay for full training.
- Full training. We train at scale with each loss under identical attack budgets and sweep the rejection cost $d$.
- Mechanism analysis. We look for the theorized fence-sitting failure directly. We measure the density of benign-borderline inputs within margin $\rho$ of the boundary before and after perturbation, for each loss. We also test whether that density predicts false refusal.
Baselines
The do-nothing reference is the unmodified LLaVA-1.5-13B, run on every benchmark with the same decoding settings and evaluator. USB reports that open-source models of this family exceed 80% attack success rate there, which sets the floor we must clear.
The strongest training-time competitor is SafeMLLM, retrained on our data mix with its published recipe and the same LoRA rank, attack steps, and GPU-hour budget as our method. Its reported numbers are an average ImgJP attack success rate of 1.50%, against 62.00% for VLGuard, and an MM-Vet score of 37.8 for LLaVA-13B after tuning. To beat SafeMLLM, we must match its ImgJP attack success within one point, stay within one point of its MM-Vet score, and lower benign false refusal on XSTest, OKTest, OR-1k, and USB harmless prompts by a margin exceeding seed variance over three seeds. No reported result was found on Lacuna for SafeMLLM's over-refusal or USB performance, so we measure those ourselves.
The strongest test-time competitor is Magic Image, reproduced on LLaVA with its dual loss and the same borderline and jailbreak pools. Its reported OKTest refusal on Llava-v1.6-mistral falls from 17.33% to 3.00%, and its GCG defense rate rises from 14.18% to 58.96%. We also report a CCAT-style gate trained with its confidence-decay target and thresholded, which isolates "any abstention region" from "a calibrated abstention region."
The key ablation is the identical gate trained with cross-entropy under the same attacks. We tune its thresholds post hoc on the same validation set, using Chow's rule on gate confidence with the same $d$. This is the ablation that isolates the loss. If it matches the shifted loss, our central claim is false.
Why not just…?
Why not just keep refusal in the text stream and harden it with SafeMLLM-style adversarial training? SafeMLLM already achieves very low attack success on the attacks it trains against, and it slightly raises MM-Vet. The open question is what happens on held-out attack families and benign borderline inputs, and for this we found no reported numbers. Adversarial training is known to overfit to its training threat model. CCAT documents a 99.4% error on unseen $\ell_2$ attacks after $\ell_\infty$ training. Our held-out FigStep, SIUO, and GCG evaluations directly test whether SafeMLLM inherits this. If SafeMLLM matches our gate on those evaluations and on over-refusal, then the gate adds nothing, and we will say so.
Why not just train a refusal classifier with cross-entropy and tune its threshold? This is the real threat. Without adversaries, convex losses with a tuned threshold are adequate for reject-option learning. The theory predicts failure only under perturbation near the boundary, and only proves it for linear models. We therefore make this the primary ablation rather than a side baseline, give it the same tuning budget, and pre-register the decision rule. The shifted loss must lower false refusal at matched held-out attack success, or the proposal's claim is refuted.
Why not just use test-time defenses like Magic Image or Self-Calibrated Consistency? Self-Calibrated Consistency is impressively cheap for zero-shot CLIP. It raises robust accuracy from 2.70% to 51.68% at a 1.3% clean cost. However, it targets classification rather than generative refusal, and its authors note its anchors collapse at $\epsilon = 8/255$. Magic Image can be overridden by an attacker's own image. We run Magic Image under an adaptive attacker who optimizes through the prepended prompt, and the same attacker is used against our gate, so the comparison is on equal terms.
Why not just filter inputs with OCR or denoising? FigStep reports that OCR and noise defenses were largely ineffective against 82.50% average attack success. SIUO cases contain no artifact to filter: 13 of 15 models there fall below a 50% safe rate. We include an OCR-plus-denoise filter only as a latency and over-refusal reference.
Proposed Experiments
Training-distribution attacks. Seen attacks are ImgJP and $\ell_\infty$ pixel attacks at $\epsilon \in \{1/255, 4/255\}$, following On the Adversarial Robustness of Multi-Modal Foundation Models, evaluated on MM-SafetyBench and AdvBench.
Held-out attacks. Unseen attacks are FigStep on SafeBench, SIUO safe-safe composites, USB-Hard across all four image/text risk combinations, and GCG suffixes. Harmfulness is judged with the HarmBench classifier, which has 93.19% human agreement, at a fixed token limit, because HarmBench shows generation length can swing attack success by up to 30%.
Utility. Utility is measured by MM-Vet and by false refusal on XSTest, OKTest, OR-1k, and USB harmless prompts. Abstention is reported as its own rate, weighted by $d$.
Decisive checks. The main result is a set of curves: benign false refusal against held-out attack success rate, traced by sweeping $d$ and threshold for every method. The shifted loss should dominate the cross-entropy gate on held-out attacks while roughly tying it on seen attacks. That pattern is the signature of a boundary effect rather than extra capacity.
Mechanism check. The count of benign-borderline inputs inside the margin under perturbation should be lower for the shifted loss, and should explain most of its false-refusal gain.
Robustness check. An adaptive white-box attack on the full gate-plus-generator pipeline must not erase the advantage.
Stopping rule. Before any large-scale training, the synthetic replication on frozen features must show the minima-jump. If it does not, we stop and report that the linear theory does not describe these representations.
References
- Theoretically Grounded Loss Functions and Algorithms for Adversarial Robustness
- Towards Calibrated Losses for Adversarial Robust Reject Option Classification
- USB: A Comprehensive and Unified Safety Evaluation Benchmark for Multimodal Large Language Models
- Towards Robust Multimodal Large Language Models Against Jailbreak Attacks
- Reimagining Safety Alignment with An Image
- Confidence-Calibrated Adversarial Training: Generalizing to Unseen Attacks
- Adversarial Robustness in One-Stage Learning-to-Defer
- The Adversarial Consistency of Surrogate Risks for Binary Classification
- Towards Consistency in Adversarial Classification
- Self-Calibrated Consistency can Fight Back for Adversarial Robustness in Vision-Language Models
- $\mathsf{FigStep}$ : Jailbreaking Large Vision-Language Models via Typographic Visual Prompts
- Safe Inputs but Unsafe Output: Benchmarking Cross-modality Safety Alignment of Large Vision-Language Models
- On the Adversarial Robustness of Multi-Modal Foundation Models
- HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal