ViDDAR: Vision Language Model-Based Task-Detrimental Content Detection for Augmented Reality

Yanming Xiu, Tim Scargill, Maria Gorlatova

Introduction

As Augmented Reality (AR) technology becomes more pervasive, the risk of "digital clutter" and malicious overlays increases. ViDDAR is a novel system designed to detect "task-detrimental" content—virtual objects that interfere with a user's ability to perform real-world tasks safely. Unlike traditional Image Quality Assessment (IQA) tools that focus on pixel-level fidelity, ViDDAR leverages the semantic reasoning of Vision Language Models (VLMs) to understand context and intent.

The Threat Model: Obstruction and Manipulation

The researchers identify two primary categories of detrimental content:

  1. Obstruction Attacks: Virtual content physically obscures critical real-world objects, such as a navigation arrow blocking a "Stop" sign.
  2. Information Manipulation Attacks: Virtual content is placed to semantically alter the user's perception of a real object, potentially leading to dangerous actions.
Figure 2: Examples of obstruction attacks where virtual overlays (b-d) block or obscure a real-world stop sign (a).

A key challenge in detection is context. A "Stop" sign is safety-critical on a road but irrelevant if it appears as a poster in a bedroom. ViDDAR uses VLMs to distinguish these nuances, identifying "key objects" based on the immediate environment and ignoring non-critical obstructions.

System Architecture

ViDDAR employs a multi-tiered user-edge-cloud architecture to balance computational intensity with real-time requirements:

  • Obstruction Detection: The system captures raw and augmented images. A cloud-based VLM identifies key objects, while an edge server utilizes Grounding DINO and the Segment Anything Model (SAM) to generate precise masks. Comparing these masks allows the system to quantify obstruction levels.
  • Information Manipulation Detection: This qualitative task evaluates alignment, style similarity, and semantic misrepresentation. For example, placing a virtual plant on a smart speaker might trick a user into watering the electronic device, causing damage.
Figure 6: An information manipulation attack where a virtual plant (b) is placed on a smart speaker (a), potentially misleading the user into treating the electronic device as a flowerpot.

Evaluation and Results

The system was evaluated using state-of-the-art VLMs, including GPT-4o and LLaVA-Next-8b, against a custom-built dataset of 420 image pairs validated by a user study.

  • Accuracy: ViDDAR achieved 92.15% accuracy in obstruction detection using GPT-4o, vastly outperforming traditional computer vision baselines like Saliency Maps (~51%). Manipulation detection reached 82.46% accuracy.
  • Latency: Obstruction detection achieved sub-second latency (533 ms), making it suitable for real-time safety alerts. Information manipulation detection currently requires ~9.6 seconds, suggesting it is better suited for background monitoring or periodic checks.

Conclusion

ViDDAR demonstrates that VLMs can effectively secure AR environments by understanding the semantic relationship between virtual and real-world objects. While latency remains a challenge for complex reasoning tasks, the system provides a robust framework for preventing digital overlays from causing physical-world harm.

Summary updated

BibTeX
@article{xiu2025viddar,
  author = {Yanming Xiu and Tim Scargill and Maria Gorlatova},
  title = {{ViDDAR: Vision Language Model-Based Task-Detrimental Content Detection for Augmented Reality}},
  journal = {IEEE Trans. Vis. Comput. Graph.},
  year = {2025},
  doi = {10.1109/tvcg.2025.3549147},
}

Loading related content...

Figure