DailySafe: Benchmarking Dynamic Safety Awareness in Everyday Multi-Turn Multimodal Interactions

Xuankun Rong, Xinhao Deng, Wenzhang Jiang, Ziheng Peng, Siyuan Liang, Wenke Huang, Pablo Arbelaez, Mang YearXiv 2026 · arXiv

As Multimodal Large Language Models (MLLMs) transition from research novelties to everyday assistants, we face a new safety frontier: Dynamic Safety Awareness. While current benchmarks excel at catching "one-shot" harmful requests (like "how do I build a bomb?"), they often miss the subtle, evolving risks that emerge during a normal conversation.

A new paper introduces DailySafe, a benchmark designed to test whether AI can connect the dots across multiple turns and images to spot implicit dangers in otherwise benign tasks.

TL;DR

DailySafe is a novel benchmark that evaluates "Dynamic Safety Awareness"—an MLLM's ability to recognize risks that emerge over multi-turn, multimodal interactions even when the user's intent remains benign. The study reveals a massive performance gap between top-tier proprietary models and open-weight alternatives, highlighting that current safety alignment often fails to account for risks hidden in conversational history and visual context.

What Problem Does This Paper Address?

Most AI safety research focuses on "static" risks: adversarial attacks or explicitly harmful instructions. However, real-world assistance is rarely one-shot. In everyday life, a request that seems perfectly safe in isolation can become life-threatening based on what happened previously.

Consider a kitchen assistant:

  • Turn 1: The user shows an image of raw chicken being prepared on a red cutting board.
  • Turn 4: The user asks, "Can I cut these tomatoes on the red board now?"

The request in Turn 4 is benign. But if the model has forgotten the "clue" from Turn 1 (raw poultry contamination), it might give a green light to a major health risk. This paper argues that true safety requires Dynamic Safety Awareness: tracking task conditions, recognizing implicit contextual risks, and providing safe alternatives without sacrificing helpfulness.

Figure 1. Motivation of DailySafe. Everyday assistance requires Dynamic Safety Awareness to recognize implicit risks across conversations, even when user intent remains benign.

Methods

The authors formalize the problem using a Safety Kernel $K = (g, r, c)$, where $g$ is a legitimate goal, $r$ is a documented risk, and $c$ are the conditions that trigger that risk.

To build the benchmark, they employed a rigorous five-stage pipeline:

  1. Task Grounding: Seeding scenarios with real human-AI dialogue topics from datasets like WildChat.
  2. Risk Grounding: Mapping these topics to authoritative safety sources (e.g., public health guidelines, product safety reports).
  3. Interaction Synthesis: Using LLMs to generate plausible multi-turn task progressions where a "critical turn" introduces a contextual risk.
  4. Visual Realization: Retrieving and editing real-world images to provide visual evidence of the evolving state.
  5. Human Verification: Ensuring every case is plausible, the risk is valid, and the history is truly necessary to identify the danger.

Evaluation is performed using an LLM-as-a-Judge (specifically Claude Opus 5.5) which uses case-specific "safety notes" to score models on two independent metrics: Safety Rate (SR) and Helpfulness Rate (HR).

Figure 2. Construction pipeline of DailySafe, which builds naturalistic multi-turn, multi-image interactions grounded in real user dia- logues and safety knowledge through interaction synthesis, visual realization, and human verification.

Datasets

The core contribution is the DailySafe benchmark:

  • 195 naturalistic cases spanning 10 everyday domains (Food, Home, Health, Care, Mobility, Work, etc.).
  • 42 task categories ranging from medication management to household maintenance.
  • 692 user turns and 475 images, with each case typically lasting 3–4 turns.
  • Grounded in authentic user data from WildChat and Anthropic’s HH-RLHF.

Key Insights and Ideas

  • The "Vision Gap": Models struggle significantly more with visual evidence than textual evidence. When images were replaced with human-written text descriptions, safety rates soared for almost all models. This suggests that the bottleneck isn't just safety reasoning, but the ability to extract safety-critical features from raw pixels.
  • Reasoning Over State, Not Just Context: The authors tested "Single-turn Consolidation"—putting all history into one big prompt. Surprisingly, this didn't consistently help. This implies that the challenge is specifically about reasoning through state changes over time, rather than just handling long-context windows.
  • Alignment Trade-offs: Many existing safety enhancements (like VLGuard) actually hurt helpfulness on DailySafe, causing models to refuse benign requests out of over-caution. Finding the balance between "don't be dangerous" and "be useful" remains a major hurdle.

Key Results

The evaluation of 16 MLLMs (including "future" models like GPT-6 and Claude 5.5) showed a dramatic performance spread:

  • Proprietary Leaders: Claude Opus 5.5 and GPT-6 Sol led the pack with Safety Rates (SR) above 92%.
  • Open-Weight Struggle: Most open-weight models lagged behind. For instance, the compact Qwen3-VL-8B-Instruct scored a dismal 2.91% SR, frequently missing risks that were clearly established in earlier turns.
  • Visual Dependence: Removing images (the "Text Only" setting) caused a massive drop in safety for top models—Gemini 3.1 Pro Preview's SR fell by 41.4 percentage points, proving these models do rely on visual cues, even if they don't always interpret them correctly.
Figure 4. Effect of visual evidence on DailySafe. Changes in mean SR relative to Full, in percentage points, when images are re- moved (Text Only, blue) or replaced with human-written descrip- tions (Image Description, orange). Removing images consistently reduces SR, whereas descriptions improve most models.

Limitations and Future Work

  • Scale: While high-quality, the 195-case dataset is relatively small. Scaling this while maintaining human-verified "realism" is a challenge.
  • Judge Reliability: While the LLM-as-a-Judge showed ~87-90% agreement with humans, it is not a perfect substitute for human intuition in complex safety scenarios.
  • Future Directions: The authors suggest that future safety alignment must move beyond "don't say bad words" toward "understand the physical consequences of actions in a changing environment." Improving visual feature extraction for safety-critical objects (like a dirty knife or a thin mattress) is a clear priority.

Summary updated

Abstract

As multimodal large language models (MLLMs) become increasingly integrated into everyday life, their ability to provide safe and reliable assistance in realistic daily activities remains insufficiently explored. Existing safety benchmarks primarily focus on isolated harmful requests or adversarial scenarios, overlooking the safety risks that may arise as ordinary tasks unfold. In practice, users pursue legitimate goals through multi-turn, multimodal interactions, where seemingly benign requests may become safety-critical when considered alongside earlier actions and visual observations. This calls for Dynamic Safety Awareness, the ability to track evolving task conditions, recognize implicit contextual risks, and provide safe and helpful assistance. To systematically evaluate this capability, we introduce DailySafe, a benchmark comprising 195 naturalistic multi-turn, multi-image interactions across 10 everyday domains and 42 task categories. Grounded in authentic user dialogue topics and documented safety knowledge, DailySafe captures realistic scenarios where safety-critical evidence is progressively revealed across conversational turns and visual observations. Using case-specific safety annotations, DailySafe assesses whether MLLMs can recognize and appropriately address evolving risks while supporting users' legitimate goals, bringing multimodal safety evaluation closer to real-world everyday AI assistance. DailySafe is publicly available at: https://github.com/XuankunRong/DailySafe.
BibTeX
@misc{rong2026dailysafe,
  author = {Xuankun Rong and Xinhao Deng and Wenzhang Jiang and Ziheng Peng and Siyuan Liang and Wenke Huang and Pablo Arbelaez and Mang Ye},
  title = {{DailySafe: Benchmarking Dynamic Safety Awareness in Everyday Multi-Turn Multimodal Interactions}},
  year = {2026},
  eprint = {2611.26298},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.26298},
}
Figure