Abstract
Researchers have developed a lightweight, glasses-mounted camera system paired with a fine-tuned Multimodal Large Language Model (MLLM) to provide people with blindness and low vision (pBLV) with enhanced spatial awareness. By fine-tuning the LLaVA model on a custom spatial reasoning dataset, the system offers real-time, voice-based guidance on object distances and navigation paths. This specialized approach outperforms general-purpose models like GPT-4 in metric-based spatial tasks, bridging the gap between descriptive AI and actionable navigational assistance.
Introduction
While contemporary AI models can describe visual scenes with high fidelity, they frequently struggle with precise spatial reasoning—the "where" and "how far" essential for daily navigation. For individuals with visual impairments, a description like "there is a pool table" is significantly less useful than "the pool table is 10 meters away at your 2 o'clock position."
Existing assistive technologies often force a compromise between bulky, specialized hardware and general-purpose AI that lacks spatial precision. This research introduces a "spatial sixth sense" via a lightweight attachment for standard eyewear, powered by a model specifically trained to understand depth and relative positioning.
System Architecture and Methods
The proposed solution consists of an integrated hardware-software stack designed for comfort and intelligence.
Hardware Implementation
The capture device is a "thumb-sized" module featuring a Seeed Studio XIAO ESP32 S3 Sense. It utilizes an OV2640 camera and Bluetooth Low Energy (BLE) 5.0 to stream visual data to a smartphone, minimizing the weight on the user's face.
Software and Model: LVS-LLaVA
A React Native mobile application serves as the user interface, handling voice-to-text and text-to-speech interactions. The core intelligence resides on a local server running LVS-LLaVA (Low Vision Spatial-LLaVA), a fine-tuned version of LLaVA-1.5.
Spatial Fine-Tuning
To instill spatial awareness, the researchers developed a novel data pipeline. They converted 2D internet-scale images into 3D point clouds using depth estimation. This allowed them to programmatically determine "ground truth" distances and positions. GPT-4 Vision was then used to generate contextually relevant, spatially-aware question-answer pairs based on this 3D data, creating a robust training set for LVS-LLaVA.
Datasets and Evaluation
The model was evaluated using two primary datasets:
- LVSQA (Low Vision Spatial Question Answering): A new benchmark containing 300 queries across 100 real-world indoor scenes, focusing on navigation, proximity, and spatial relationships.
- VizWiz: A standard VQA dataset for the visually impaired, used to ensure the model retained general reasoning capabilities.
Key Results
The fine-tuned LVS-LLaVA model demonstrated significant improvements over baseline models:
- Accuracy: On the LVSQA benchmark, the model outperformed both the base LLaVA and GPT-4. It achieved a BLEU-1 score of 0.433, notably higher than GPT-4’s 0.324.
- Distance Estimation: Real-world testing showed that most distance estimations were accurate within a 5-meter range, providing actionable information for users.
- Generalization: Performance on the VizWiz dataset remained stable (0.618 BLEU-1 for LVS-LLaVA vs. 0.650 for base LLaVA), indicating that spatial specialization does not cause "catastrophic forgetting" of general visual knowledge.
Conclusion and Future Work
This work demonstrates that spatial reasoning is a specialized skill requiring targeted fine-tuning. By offloading heavy computation to a server while maintaining a minimal hardware footprint, the authors provide a viable path for practical assistive wearables. Future efforts will focus on increasing distance precision for higher-stakes navigation and refining the user interface for long-term daily use.
Summary updated