WearVQA: A Visual Question Answering Benchmark for Wearables in Egocentric Authentic Real-world scenarios

Eun Chang, Zhuangqun Huang, Yiwei Liao, Sagar Ravi Bhavsar, Amogh Param, Tammy Stark, Adel Ahmadyan, Xiao Yang, Jiaqi Wang, Ahsan Abdullah, Giang Nguyen, Akil Iyer, David Hall, Elissa Li, Shane Moon, Nicolas Scheffer, Kirmani Ahmed, Babak Damavandi, Rakesh Wanga, Anuj Kumar, Rohit Patel, Xin Luna Dong

TL;DR

WearVQA is a new visual question answering benchmark specifically designed for wearable AI assistants, featuring 2,520 egocentric image-question-answer triplets. Unlike traditional datasets with pristine, third-person photos, WearVQA captures real-world image degradations (such as blur, low light, and occlusions) and complex reasoning tasks typical of smart glass users. Evaluation reveals that even state-of-the-art multimodal LLMs struggle significantly in these authentic environments, scoring between 23% and 52% accuracy.

What Problem Does This Paper Address?

Most existing Visual Question Answering (VQA) benchmarks rely on high-quality, well-framed, third-person imagery. However, wearable devices like smart glasses operate in an egocentric (first-person) paradigm. This introduces severe visual artifacts: motion blur from head movements, poor or uneven lighting, unzoomed wide-angle views, and frequent occlusions from the user's own hands.

Furthermore, user queries to wearable assistants are inherently contextual and assume a shared point of view (e.g., "How do I use the tool on my left?"). This paper bridges the gap between curated lab datasets and the messy reality of consumer wearables by introducing a benchmark that tests models on authentic, degraded egocentric data.

Methods

The authors collected real-world egocentric data using Ray-Ban Meta smart glasses. To capture realistic motion blur and framing issues, they sampled frames directly from recorded videos of daily activities. To ensure the benchmark remains challenging and "forward-looking," they filtered out any questions that current state-of-the-art (SOTA) multimodal models could consistently answer correctly.

For evaluation, they developed an automated "LLM-as-a-judge" framework using GPT-4o. The judge evaluates model responses across five distinct criteria: factual correctness, relevance, completeness, egocentric phrasing, and conciseness. This automated pipeline achieved a remarkable 96% alignment accuracy when cross-checked against human annotations.

Datasets

The paper introduces WearVQA, a dataset containing 2,520 carefully curated image-question-answer triplets. It is split into a public test set (1,500 pairs) and a private test set (1,000 pairs).

The dataset is highly diverse, spanning:

  • 7 Domains: Text/documents, food/drinks, landmarks/travel, shopping/products, gardening/plants, animals/pets, and hobbies/activities.
  • 10 Cognitive Tasks: Ranging from basic text/image recognition to complex multi-step reasoning, spatial inference, and mathematical calculations.
  • 6 Image Quality Issues: Blurred, cut-off, low light, unmagnified (lack of zoom), occluded, and rotated. Notably, 54% of the images contain at least one of these quality issues, and 42% feature hand-holding or finger-pointing.
Figure 1: Example image-question-answer triples across 10 visual cognitive task types in the WearVQA benchmark.

Key Insights and Ideas

  • The Wide-Angle Penalty: "Unzoomed" or wide-angle images represent the single biggest hurdle for multimodal LLMs. Fixed-focal-length wearable cameras often capture small text or distant objects, causing model accuracy to drop because they cannot resolve fine details.
  • The Low-Light Paradox: Counterintuitively, low-light conditions had the least negative impact on QA accuracy. The authors hypothesize that darkness naturally acts as a background subtractor, masking distracting elements and leaving the primary subject prominently illuminated under focused light.
  • Reasoning vs. Recognition: Models are relatively decent at basic visual recognition but fail catastrophically at tasks requiring spatial reasoning, object counting, or mathematical calculations based on visual text.
Examples of egocentric image quality issues

Key Results

  • Overall performance across all tested models is surprisingly low, ranging from 23.9% to 51.5% accuracy, highlighting the difficulty of the benchmark.
  • GPT-4o emerged as the top-performing model with an overall accuracy of 51.5%, followed closely by Gemini-1.5-Pro (45.4%) and the open-source Qwen2.5-VL-72B-Instruct (45.1%).
  • Image quality degradation causes a substantial performance drop across the board, reducing accuracy by 5% to 16% compared to high-quality images.
  • On specific reasoning tasks like Object Counting and Spatial Reasoning, top-1 accuracy dropped to a dismal 38% and 41% respectively.

Limitations and Future Work

  • No External Knowledge (RAG): The dataset explicitly restricts questions to those answerable using only the image and common sense. Real-world wearable assistants frequently require Multimodal Retrieval-Augmented Generation (MM-RAG) to answer queries about specific products or obscure facts.
  • Language Constraint: The benchmark is currently limited to English-language queries.
  • Static Images: While the images are derived from video frames, the benchmark evaluates static single-frame VQA, missing out on temporal dynamics, fluctuating lighting conditions, and video-level reasoning.

The authors plan to expand WearVQA to include multilingual support and dynamic video sequences in future iterations.

Summary updated

Abstract

We introduce WearVQA, the first benchmark specifically designed to evaluate the Visual Question Answering (VQA) capabilities of multi-model AI assistant on wearable devices like smart glasses. Unlike prior benchmarks that focus on high-quality, third-person imagery, WearVQA reflects the unique challenges of ego-centric interaction-where visual inputs may be occluded, poorly lit, unzoomed, or blurry, and questions are grounded in realistic wearable use cases. The benchmark comprises 2,520 carefully curated image-question-answer triplets, spanning 7 diverse image domains including both text-centric and general scenes, 10 cognitive task types ranging from basic recognition to various forms of reasoning, and 6 common wearables-specific image quality issues. All questions are designed to be answerable using only the visual input and common senses. WearVQA is paired with a rigorous LLM-as-a-judge evaluation framework with 96% labeling accuracy. Open-source and proprietary multi-model LLMs achieved a QA accuracy as low as 24-52% on WearVQA, with substantial drops on lower-quality images and reasoning-heavy tasks. These observations position WearVQA as a comprehensive and challenging benchmark for guiding technical advancement towards robust, real-world multi-model wearables AI systems.
BibTeX
@inproceedings{chang2025wearvqa,
  author = {Eun Chang and Zhuangqun Huang and Yiwei Liao and Sagar Ravi Bhavsar and Amogh Param and Tammy Stark and Adel Ahmadyan and Xiao Yang and Jiaqi Wang and Ahsan Abdullah and Giang Nguyen and Akil Iyer and David Hall and Elissa Li and Shane Moon and Nicolas Scheffer and Kirmani Ahmed and Babak Damavandi and Rakesh Wanga and Anuj Kumar and Rohit Patel and Xin Luna Dong},
  title = {{WearVQA: A Visual Question Answering Benchmark for Wearables in Egocentric Authentic Real-world scenarios}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=s5p9ByKN1j},
}
Figure