Introduction
Visual navigation for the visually impaired presents a significant challenge due to the unpredictable nature of urban environments. Traditional computer vision systems often rely on "closed-set" object detection, which limits their ability to recognize unusual but hazardous obstacles—the so-called "long-tail" problem. While Multimodal Large Language Models (LLMs) like GPT-4V offer powerful zero-shot reasoning capabilities, their high computational cost and latency have historically made them unsuitable for real-time safety applications. VisionGPT addresses this by combining the speed of open-vocabulary object detection with the reasoning power of LLMs to provide a low-latency, intelligent navigation aid.
System Architecture
The VisionGPT framework is designed for deployment on mobile edge devices, utilizing a multi-module approach to balance speed and intelligence. At its core, the system employs YOLO-World, an open-vocabulary detector that can identify any object described in text without retraining. This is managed by a Detection Class Manager, which allows the LLM to dynamically update the detector's focus based on the user's environment (e.g., switching from traffic lights on a street to benches in a park).
As shown in the architecture, the "Master Program" processes raw object information through an Image Processor and an Anomaly Handle module. The system uses a hybrid LLM approach: GPT-3.5 handles low-level data formatting to maintain low latency, while GPT-4 provides high-level reasoning for complex scene descriptions. To further optimize performance, the system implements frame compensation, increasing the effective processing speed from 16 FPS to 73 FPS.
Spatial Heuristics and Anomaly Detection
A key innovation of VisionGPT is its definition of "anomalies" as contextual hazards rather than just rare visual artifacts. This is operationalized through the Type H image splitter, a spatial heuristic that divides the camera frame into four distinct zones to prioritize threats.
As illustrated in the spatial visualization:
- Zones 1 & 2 (Left/Right): Monitor for fast-moving side hazards like cyclists or cars.
- Zone 4 (Front/Top Center): Identifies distant objects for long-term path planning.
- Zone 3 (Ground/Bottom Center): The most critical area. Any object detected here is immediately flagged as a potential trip hazard or anomaly, triggering an emergency response.
Experimental Evaluation
The researchers evaluated VisionGPT using a custom dataset of 50 first-person video clips covering over 100,000 detected objects across urban, suburban, and park environments. The system demonstrated robust performance in various weather conditions, including sun, cloud, and rain.
Key findings from the evaluation include:
- Real-Time Performance: The system achieved an average end-to-end latency of 60ms on mobile hardware (M2 chip/iPhone 14 Pro Max).
- High Accuracy: In zero-shot anomaly detection, the framework reached an Average Precision (AP) of 88.01 and an AUC of 87.26.
- Prompt Optimization: Interestingly, "Low Sensitivity" prompts—which instruct the LLM to report only imminent threats—yielded the highest accuracy, helping to prevent user alert fatigue.
Conclusion and Future Work
VisionGPT successfully bridges the gap between high-level LLM reasoning and real-time edge computing for visual navigation. By utilizing dynamic vocabularies and spatial heuristics, it provides a flexible and accurate safety tool for the visually impaired. Future work will focus on reducing cloud dependency to ensure reliability in areas with poor connectivity and refining spatial reasoning through 3D depth estimation to further reduce false positives.
Summary updated