Evaluating the safety of Multimodal Large Language Models (MLLMs) is a high-stakes game. We rely on situational-safety benchmarks to tell us if a model can distinguish between a safe rail yard and the danger of standing on the tracks. But what are these benchmarks actually measuring?
A new paper, "What Does a Safety Label Measure?", reveals that many of our standard safety tests are "leaky." Models often achieve high scores not by understanding the scene, but by exploiting data shortcuts like image resolution or textual cues in the request.
TL;DR
This paper introduces the Benchmark Card to audit situational-safety benchmarks for "shortcuts" (like image size or text-only cues) that allow models to guess safety labels without visual reasoning. It also proposes the Spot-Stop decomposition, a framework that explains why models are often better at ranking risks than labeling them, and shows that simple "recentering" of decision thresholds can significantly improve safety performance.
What Problem Does This Paper Address?
When an MLLM labels a scene as "UNSAFE," we assume it has "seen" the hazard. However, benchmarks can contain hidden signals. For example, if all "unsafe" images in a dataset are high-resolution web scrapes while "safe" images are low-res snapshots, a model might just learn that "big image = danger."
Furthermore, there is a persistent gap between ranking and labeling. As shown in Figure 1, models like Qwen3.5 and LFM2.5 correctly identify that standing on a track is riskier than looking at a rail yard from above (ranking), yet they might still label both scenes as "SAFE" in their final answer (labeling).
Methods
The authors tackle these issues with two main contributions:
- The Benchmark Card: A diagnostic tool consisting of seven fields (see Figure 6/Table 1) that measure how much of a safety label can be recovered from file metadata, the text request alone, or the scene itself.
- Spot-Stop Decomposition: A mathematical framework that splits the performance gap between ranking and labeling into three parts:
- Threshold part: The model's internal "cutoff" for what constitutes "unsafe" is poorly calibrated.
- Overlap part: The model's internal scores for safe and unsafe scenes are too similar to separate.
- Within-contrast part: The model only understands the risk when comparing two related scenes side-by-side.
The authors also test label-free fixes, such as "recentering" the model's decision threshold based on the average score of a batch of queries.
Datasets
The study audits six major situational-safety benchmarks:
- MSSBench (Chat and Embodied)
- VSCBench
- ASIMOV
- MOSSBench + SIUO
- HoliSafe
The authors also release a corrected MSSBench chat split specifically designed to remove image-size shortcuts.
Key Insights and Ideas
- The "Size Rule" is surprisingly powerful: In the original MSSBench chat, a simple rule—"if the longest side is > 640 pixels, it's UNSAFE"—achieves 82.3% accuracy. This beats or ties six out of seven commercial models, including GPT-4 variants.
- Textual "Leakage": In benchmarks like HoliSafe, the text request alone is so descriptive that models can rank safety pairs with over 90% AUROC without ever looking at the image.
- Ranking > Labeling: Most of the 14 open-source MLLMs tested are significantly better at spotting which scene is more dangerous than they are at actually labeling a dangerous scene as "UNSAFE."
- Thresholds are the main culprit: A large portion of safety failures (32-45% for top models) isn't due to a lack of understanding, but simply a poorly placed decision threshold.
Key Results
The audit results, summarized in the Benchmark Card (Figure 6), show varying levels of "leakage" across datasets.
- Spot-Stop Gap: Figure 5 illustrates how the gap between ranking (R) and labeling (A) varies by model. For many models, the "threshold part" (dark gray) is the largest fixable component.
- Recentering Works: By simply adjusting the threshold to the batch mean, the authors improved balanced accuracy on VSCBench for 13 out of 14 models.
- Corrected Split: On the authors' new size-controlled MSSBench split, the "size rule" falls to chance, forcing models to actually rely on visual situational reasoning.
Limitations and Future Work
- Model Scale: The study focused on models between 0.5B and 9.7B parameters. It remains to be seen if massive frontier models (like GPT-4o or Gemini 1.5 Pro) exhibit the same "Spot-Stop" patterns.
- Batch Dependency: The "recentering" fix requires a batch of queries or reference scenes to work, which might not be available in every real-time deployment scenario.
- Free-form vs. Logits: The decomposition relies on teacher-forced log-probabilities. While these correlate with free-form answers, they don't perfectly capture the nuances of a model's generated reasoning.
The authors suggest that future benchmark builders should use their Benchmark Card as a checklist to ensure they are measuring visual safety, not just dataset artifacts.
Summary updated