VisionGPT: LLM-Assisted Real-Time Anomaly Detection for Safe Visual Navigation

Hao Wang, Jiayou Qin, Ashish Bastola, Xiwen Chen, John Suchanek, Zihao Gong, Abolfazl RaziarXiv 2024 · DOI · arXiv

Introduction

Visual navigation for the visually impaired presents a significant challenge due to the unpredictable nature of urban environments. Traditional computer vision systems often rely on "closed-set" object detection, which limits their ability to recognize unusual but hazardous obstacles—the so-called "long-tail" problem. While Multimodal Large Language Models (LLMs) like GPT-4V offer powerful zero-shot reasoning capabilities, their high computational cost and latency have historically made them unsuitable for real-time safety applications. VisionGPT addresses this by combining the speed of open-vocabulary object detection with the reasoning power of LLMs to provide a low-latency, intelligent navigation aid.

System Architecture

The VisionGPT framework is designed for deployment on mobile edge devices, utilizing a multi-module approach to balance speed and intelligence. At its core, the system employs YOLO-World, an open-vocabulary detector that can identify any object described in text without retraining. This is managed by a Detection Class Manager, which allows the LLM to dynamically update the detector's focus based on the user's environment (e.g., switching from traffic lights on a street to benches in a park).

VisionGPT System Architecture

As shown in the architecture, the "Master Program" processes raw object information through an Image Processor and an Anomaly Handle module. The system uses a hybrid LLM approach: GPT-3.5 handles low-level data formatting to maintain low latency, while GPT-4 provides high-level reasoning for complex scene descriptions. To further optimize performance, the system implements frame compensation, increasing the effective processing speed from 16 FPS to 73 FPS.

Spatial Heuristics and Anomaly Detection

A key innovation of VisionGPT is its definition of "anomalies" as contextual hazards rather than just rare visual artifacts. This is operationalized through the Type H image splitter, a spatial heuristic that divides the camera frame into four distinct zones to prioritize threats.

The Type H image splitter for spatial anomaly detection

As illustrated in the spatial visualization:

  • Zones 1 & 2 (Left/Right): Monitor for fast-moving side hazards like cyclists or cars.
  • Zone 4 (Front/Top Center): Identifies distant objects for long-term path planning.
  • Zone 3 (Ground/Bottom Center): The most critical area. Any object detected here is immediately flagged as a potential trip hazard or anomaly, triggering an emergency response.

Experimental Evaluation

The researchers evaluated VisionGPT using a custom dataset of 50 first-person video clips covering over 100,000 detected objects across urban, suburban, and park environments. The system demonstrated robust performance in various weather conditions, including sun, cloud, and rain.

Key findings from the evaluation include:

  • Real-Time Performance: The system achieved an average end-to-end latency of 60ms on mobile hardware (M2 chip/iPhone 14 Pro Max).
  • High Accuracy: In zero-shot anomaly detection, the framework reached an Average Precision (AP) of 88.01 and an AUC of 87.26.
  • Prompt Optimization: Interestingly, "Low Sensitivity" prompts—which instruct the LLM to report only imminent threats—yielded the highest accuracy, helping to prevent user alert fatigue.

Conclusion and Future Work

VisionGPT successfully bridges the gap between high-level LLM reasoning and real-time edge computing for visual navigation. By utilizing dynamic vocabularies and spatial heuristics, it provides a flexible and accurate safety tool for the visually impaired. Future work will focus on reducing cloud dependency to ensure reliability in areas with poor connectivity and refining spatial reasoning through 3D depth estimation to further reduce false positives.

Summary updated

Abstract

This paper explores the potential of Large Language Models(LLMs) in zero-shot anomaly detection for safe visual navigation. With the assistance of the state-of-the-art real-time open-world object detection model Yolo-World and specialized prompts, the proposed framework can identify anomalies within camera-captured frames that include any possible obstacles, then generate concise, audio-delivered descriptions emphasizing abnormalities, assist in safe visual navigation in complex circumstances. Moreover, our proposed framework leverages the advantages of LLMs and the open-vocabulary object detection model to achieve the dynamic scenario switch, which allows users to transition smoothly from scene to scene, which addresses the limitation of traditional visual navigation. Furthermore, this paper explored the performance contribution of different prompt components, provided the vision for future improvement in visual accessibility, and paved the way for LLMs in video anomaly detection and vision-language understanding.
BibTeX
@misc{wang2024visiongpt,
  author = {Hao Wang and Jiayou Qin and Ashish Bastola and Xiwen Chen and John Suchanek and Zihao Gong and Abolfazl Razi},
  title = {{VisionGPT: LLM-Assisted Real-Time Anomaly Detection for Safe Visual Navigation}},
  year = {2024},
  eprint = {2403.12415},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2403.12415},
}
Figure