What Does a Safety Label Measure? Auditing Situational-Safety Benchmarks for Multimodal LLMs

Wayner Barrios, Kenneth BarriosarXiv 2026 · arXiv

Evaluating the safety of Multimodal Large Language Models (MLLMs) is a high-stakes game. We rely on situational-safety benchmarks to tell us if a model can distinguish between a safe rail yard and the danger of standing on the tracks. But what are these benchmarks actually measuring?

A new paper, "What Does a Safety Label Measure?", reveals that many of our standard safety tests are "leaky." Models often achieve high scores not by understanding the scene, but by exploiting data shortcuts like image resolution or textual cues in the request.

TL;DR

This paper introduces the Benchmark Card to audit situational-safety benchmarks for "shortcuts" (like image size or text-only cues) that allow models to guess safety labels without visual reasoning. It also proposes the Spot-Stop decomposition, a framework that explains why models are often better at ranking risks than labeling them, and shows that simple "recentering" of decision thresholds can significantly improve safety performance.

What Problem Does This Paper Address?

When an MLLM labels a scene as "UNSAFE," we assume it has "seen" the hazard. However, benchmarks can contain hidden signals. For example, if all "unsafe" images in a dataset are high-resolution web scrapes while "safe" images are low-res snapshots, a model might just learn that "big image = danger."

Furthermore, there is a persistent gap between ranking and labeling. As shown in Figure 1, models like Qwen3.5 and LFM2.5 correctly identify that standing on a track is riskier than looking at a rail yard from above (ranking), yet they might still label both scenes as "SAFE" in their final answer (labeling).

Figure 1: Models correctly rank the track scene as riskier, but fail to flag it as UNSAFE in their final answer.

Methods

The authors tackle these issues with two main contributions:

  1. The Benchmark Card: A diagnostic tool consisting of seven fields (see Figure 6/Table 1) that measure how much of a safety label can be recovered from file metadata, the text request alone, or the scene itself.
  2. Spot-Stop Decomposition: A mathematical framework that splits the performance gap between ranking and labeling into three parts:
  3. Threshold part: The model's internal "cutoff" for what constitutes "unsafe" is poorly calibrated.
  4. Overlap part: The model's internal scores for safe and unsafe scenes are too similar to separate.
  5. Within-contrast part: The model only understands the risk when comparing two related scenes side-by-side.

The authors also test label-free fixes, such as "recentering" the model's decision threshold based on the average score of a batch of queries.

Datasets

The study audits six major situational-safety benchmarks:

  • MSSBench (Chat and Embodied)
  • VSCBench
  • ASIMOV
  • MOSSBench + SIUO
  • HoliSafe

The authors also release a corrected MSSBench chat split specifically designed to remove image-size shortcuts.

Key Insights and Ideas

  • The "Size Rule" is surprisingly powerful: In the original MSSBench chat, a simple rule—"if the longest side is > 640 pixels, it's UNSAFE"—achieves 82.3% accuracy. This beats or ties six out of seven commercial models, including GPT-4 variants.
  • Textual "Leakage": In benchmarks like HoliSafe, the text request alone is so descriptive that models can rank safety pairs with over 90% AUROC without ever looking at the image.
  • Ranking > Labeling: Most of the 14 open-source MLLMs tested are significantly better at spotting which scene is more dangerous than they are at actually labeling a dangerous scene as "UNSAFE."
  • Thresholds are the main culprit: A large portion of safety failures (32-45% for top models) isn't due to a lack of understanding, but simply a poorly placed decision threshold.

Key Results

The audit results, summarized in the Benchmark Card (Figure 6), show varying levels of "leakage" across datasets.

Table 1. Benchmark Cards for six situational-safety benchmarks. One column per benchmark, one row per field (Section 2). File-only rule, metadata classifier: accuracy or AUROC of a reader that sees only the image file. Request only: AUROC with the image removed (best / median of the fourteen models). Situation share: median intraclass correlation at the situation level. R −A > 0: models whose ranking rate R exceeds their direct-answer accuracy A with a 95% interval above zero (Section 3); in parentheses, models whose excess also exceeds g(AUROC). Threshold / overlap / within: median split of that excess, in points. Recovery: median share of the threshold part that the batch-mean threshold recovers at 50% and 10% unsafe. No pairs: no safe/unsafe contrasts. Shared request or image: 0.500 by construction. Corrected split: the MSSBench chat split we release (Section 4).
  • Spot-Stop Gap: Figure 5 illustrates how the gap between ranking (R) and labeling (A) varies by model. For many models, the "threshold part" (dark gray) is the largest fixable component.
  • Recentering Works: By simply adjusting the threshold to the batch mean, the authors improved balanced accuracy on VSCBench for 13 out of 14 models.
  • Corrected Split: On the authors' new size-controlled MSSBench split, the "size rule" falls to chance, forcing models to actually rely on visual situational reasoning.
Figure 2. The Spot-Stop gap has three parts, and their mix varies by model (MSSBench chat, 600 contrasts; rows sorted by chat AUROC, shown in parentheses). (a) R −A in points, split as in Eq. (3): threshold part (dark), overlap (gray) and order within a pair (blue). (b) Bars: the threshold part, the most any shared threshold can recover. Circles: what batch calibration recovers without labels (negative where it hurts). Above the dashed line (AUROC 0.70) it nears the bar; below, it hurts three of five models.

Limitations and Future Work

  • Model Scale: The study focused on models between 0.5B and 9.7B parameters. It remains to be seen if massive frontier models (like GPT-4o or Gemini 1.5 Pro) exhibit the same "Spot-Stop" patterns.
  • Batch Dependency: The "recentering" fix requires a batch of queries or reference scenes to work, which might not be available in every real-time deployment scenario.
  • Free-form vs. Logits: The decomposition relies on teacher-forced log-probabilities. While these correlate with free-form answers, they don't perfectly capture the nuances of a model's generated reasoning.

The authors suggest that future benchmark builders should use their Benchmark Card as a checklist to ensure they are measuring visual safety, not just dataset artifacts.

Summary updated

Abstract

Situational-safety benchmarks label each scene SAFE or UNSAFE and score a multimodal large language model (MLLM) against that label, usually by judging its free-form answer, and these scores guide deployment. We ask what such a label measures. A label score can come from the scene, the request or the benchmark's own files. We propose the Benchmark Card, seven measured fields that tell them apart, and fill it for six benchmarks. Only MSSBench chat leaks the label through its image files: image size alone reaches 82.3% accuracy, matching or beating six of seven commercial models. Elsewhere the signal outside the scene comes from design. In MOSSBench the item partition fixes the label, and on HoliSafe the request alone ranks pairs about as well as the scene. On our corrected chat split, the size rule falls to chance. We then introduce the Spot-Stop decomposition, which compares ranking with labelling on pairs of a safe and an unsafe scene. It splits the gap above a label-swap null into three parts: a threshold part, overlap between safe and unsafe scores, and order that holds only within a pair. Most of fourteen open MLLMs rank better than they label, beyond what their AUROC predicts. On chat, one shared threshold repairs a third to a half of the excess, and label-free recentering recovers it on balanced batches. Recentering on unlabelled scenes of the same situation improves ranking on VSCBench for 13 of 14 models.
BibTeX
@misc{barrios2026what,
  author = {Wayner Barrios and Kenneth Barrios},
  title = {{What Does a Safety Label Measure? Auditing Situational-Safety Benchmarks for Multimodal LLMs}},
  year = {2026},
  eprint = {2611.01125},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.01125},
}
Figure