As Multimodal Large Language Models (MLLMs) transition from research novelties to everyday assistants, we face a new safety frontier: Dynamic Safety Awareness. While current benchmarks excel at catching "one-shot" harmful requests (like "how do I build a bomb?"), they often miss the subtle, evolving risks that emerge during a normal conversation.
A new paper introduces DailySafe, a benchmark designed to test whether AI can connect the dots across multiple turns and images to spot implicit dangers in otherwise benign tasks.
TL;DR
DailySafe is a novel benchmark that evaluates "Dynamic Safety Awareness"—an MLLM's ability to recognize risks that emerge over multi-turn, multimodal interactions even when the user's intent remains benign. The study reveals a massive performance gap between top-tier proprietary models and open-weight alternatives, highlighting that current safety alignment often fails to account for risks hidden in conversational history and visual context.
What Problem Does This Paper Address?
Most AI safety research focuses on "static" risks: adversarial attacks or explicitly harmful instructions. However, real-world assistance is rarely one-shot. In everyday life, a request that seems perfectly safe in isolation can become life-threatening based on what happened previously.
Consider a kitchen assistant:
- Turn 1: The user shows an image of raw chicken being prepared on a red cutting board.
- Turn 4: The user asks, "Can I cut these tomatoes on the red board now?"
The request in Turn 4 is benign. But if the model has forgotten the "clue" from Turn 1 (raw poultry contamination), it might give a green light to a major health risk. This paper argues that true safety requires Dynamic Safety Awareness: tracking task conditions, recognizing implicit contextual risks, and providing safe alternatives without sacrificing helpfulness.
Methods
The authors formalize the problem using a Safety Kernel $K = (g, r, c)$, where $g$ is a legitimate goal, $r$ is a documented risk, and $c$ are the conditions that trigger that risk.
To build the benchmark, they employed a rigorous five-stage pipeline:
- Task Grounding: Seeding scenarios with real human-AI dialogue topics from datasets like WildChat.
- Risk Grounding: Mapping these topics to authoritative safety sources (e.g., public health guidelines, product safety reports).
- Interaction Synthesis: Using LLMs to generate plausible multi-turn task progressions where a "critical turn" introduces a contextual risk.
- Visual Realization: Retrieving and editing real-world images to provide visual evidence of the evolving state.
- Human Verification: Ensuring every case is plausible, the risk is valid, and the history is truly necessary to identify the danger.
Evaluation is performed using an LLM-as-a-Judge (specifically Claude Opus 5.5) which uses case-specific "safety notes" to score models on two independent metrics: Safety Rate (SR) and Helpfulness Rate (HR).
Datasets
The core contribution is the DailySafe benchmark:
- 195 naturalistic cases spanning 10 everyday domains (Food, Home, Health, Care, Mobility, Work, etc.).
- 42 task categories ranging from medication management to household maintenance.
- 692 user turns and 475 images, with each case typically lasting 3–4 turns.
- Grounded in authentic user data from WildChat and Anthropic’s HH-RLHF.
Key Insights and Ideas
- The "Vision Gap": Models struggle significantly more with visual evidence than textual evidence. When images were replaced with human-written text descriptions, safety rates soared for almost all models. This suggests that the bottleneck isn't just safety reasoning, but the ability to extract safety-critical features from raw pixels.
- Reasoning Over State, Not Just Context: The authors tested "Single-turn Consolidation"—putting all history into one big prompt. Surprisingly, this didn't consistently help. This implies that the challenge is specifically about reasoning through state changes over time, rather than just handling long-context windows.
- Alignment Trade-offs: Many existing safety enhancements (like VLGuard) actually hurt helpfulness on DailySafe, causing models to refuse benign requests out of over-caution. Finding the balance between "don't be dangerous" and "be useful" remains a major hurdle.
Key Results
The evaluation of 16 MLLMs (including "future" models like GPT-6 and Claude 5.5) showed a dramatic performance spread:
- Proprietary Leaders: Claude Opus 5.5 and GPT-6 Sol led the pack with Safety Rates (SR) above 92%.
- Open-Weight Struggle: Most open-weight models lagged behind. For instance, the compact Qwen3-VL-8B-Instruct scored a dismal 2.91% SR, frequently missing risks that were clearly established in earlier turns.
- Visual Dependence: Removing images (the "Text Only" setting) caused a massive drop in safety for top models—Gemini 3.1 Pro Preview's SR fell by 41.4 percentage points, proving these models do rely on visual cues, even if they don't always interpret them correctly.
Limitations and Future Work
- Scale: While high-quality, the 195-case dataset is relatively small. Scaling this while maintaining human-verified "realism" is a challenge.
- Judge Reliability: While the LLM-as-a-Judge showed ~87-90% agreement with humans, it is not a perfect substitute for human intuition in complex safety scenarios.
- Future Directions: The authors suggest that future safety alignment must move beyond "don't say bad words" toward "understand the physical consequences of actions in a changing environment." Improving visual feature extraction for safety-critical objects (like a dirty knife or a thin mattress) is a clear priority.
Summary updated