TL;DR
WearVQA is a new visual question answering benchmark specifically designed for wearable AI assistants, featuring 2,520 egocentric image-question-answer triplets. Unlike traditional datasets with pristine, third-person photos, WearVQA captures real-world image degradations (such as blur, low light, and occlusions) and complex reasoning tasks typical of smart glass users. Evaluation reveals that even state-of-the-art multimodal LLMs struggle significantly in these authentic environments, scoring between 23% and 52% accuracy.
What Problem Does This Paper Address?
Most existing Visual Question Answering (VQA) benchmarks rely on high-quality, well-framed, third-person imagery. However, wearable devices like smart glasses operate in an egocentric (first-person) paradigm. This introduces severe visual artifacts: motion blur from head movements, poor or uneven lighting, unzoomed wide-angle views, and frequent occlusions from the user's own hands.
Furthermore, user queries to wearable assistants are inherently contextual and assume a shared point of view (e.g., "How do I use the tool on my left?"). This paper bridges the gap between curated lab datasets and the messy reality of consumer wearables by introducing a benchmark that tests models on authentic, degraded egocentric data.
Methods
The authors collected real-world egocentric data using Ray-Ban Meta smart glasses. To capture realistic motion blur and framing issues, they sampled frames directly from recorded videos of daily activities. To ensure the benchmark remains challenging and "forward-looking," they filtered out any questions that current state-of-the-art (SOTA) multimodal models could consistently answer correctly.
For evaluation, they developed an automated "LLM-as-a-judge" framework using GPT-4o. The judge evaluates model responses across five distinct criteria: factual correctness, relevance, completeness, egocentric phrasing, and conciseness. This automated pipeline achieved a remarkable 96% alignment accuracy when cross-checked against human annotations.
Datasets
The paper introduces WearVQA, a dataset containing 2,520 carefully curated image-question-answer triplets. It is split into a public test set (1,500 pairs) and a private test set (1,000 pairs).
The dataset is highly diverse, spanning:
- 7 Domains: Text/documents, food/drinks, landmarks/travel, shopping/products, gardening/plants, animals/pets, and hobbies/activities.
- 10 Cognitive Tasks: Ranging from basic text/image recognition to complex multi-step reasoning, spatial inference, and mathematical calculations.
- 6 Image Quality Issues: Blurred, cut-off, low light, unmagnified (lack of zoom), occluded, and rotated. Notably, 54% of the images contain at least one of these quality issues, and 42% feature hand-holding or finger-pointing.
Key Insights and Ideas
- The Wide-Angle Penalty: "Unzoomed" or wide-angle images represent the single biggest hurdle for multimodal LLMs. Fixed-focal-length wearable cameras often capture small text or distant objects, causing model accuracy to drop because they cannot resolve fine details.
- The Low-Light Paradox: Counterintuitively, low-light conditions had the least negative impact on QA accuracy. The authors hypothesize that darkness naturally acts as a background subtractor, masking distracting elements and leaving the primary subject prominently illuminated under focused light.
- Reasoning vs. Recognition: Models are relatively decent at basic visual recognition but fail catastrophically at tasks requiring spatial reasoning, object counting, or mathematical calculations based on visual text.
Key Results
- Overall performance across all tested models is surprisingly low, ranging from 23.9% to 51.5% accuracy, highlighting the difficulty of the benchmark.
- GPT-4o emerged as the top-performing model with an overall accuracy of 51.5%, followed closely by Gemini-1.5-Pro (45.4%) and the open-source Qwen2.5-VL-72B-Instruct (45.1%).
- Image quality degradation causes a substantial performance drop across the board, reducing accuracy by 5% to 16% compared to high-quality images.
- On specific reasoning tasks like Object Counting and Spatial Reasoning, top-1 accuracy dropped to a dismal 38% and 41% respectively.
Limitations and Future Work
- No External Knowledge (RAG): The dataset explicitly restricts questions to those answerable using only the image and common sense. Real-world wearable assistants frequently require Multimodal Retrieval-Augmented Generation (MM-RAG) to answer queries about specific products or obscure facts.
- Language Constraint: The benchmark is currently limited to English-language queries.
- Static Images: While the images are derived from video frames, the benchmark evaluates static single-frame VQA, missing out on temporal dynamics, fluctuating lighting conditions, and video-level reasoning.
The authors plan to expand WearVQA to include multilingual support and dynamic video sequences in future iterations.
Summary updated