Computer Vision-Based Assistance System for the Visually Impaired Using Mobile Edge Artificial Intelligence

Jagadish K. Mahendran, Daniel T. Barry, Anita K. Nivedha, Suchendra M. BhandarkarWorkshops @ CVPR 2021 · DOI

Introduction

Navigating the modern urban landscape presents significant challenges for the visually impaired. While traditional aids like white canes are effective for ground-level obstacles, they cannot detect "hanging" hazards such as low tree branches or open windows. Conversely, while GPS-based mobile applications assist with route planning, they lack the real-time obstacle avoidance necessary for safe movement. Existing high-tech solutions often require heavy, power-hungry GPUs, leading to poor portability and short battery life. This research introduces a wearable, real-time visual assistance system that leverages edge AI to provide a portable, cost-effective, and accurate solution.

Problem Statement

The core challenge lies in bridging the gap between sophisticated deep learning models and the constraints of mobile hardware. Traditional computer vision often fails in variable real-world lighting and environments, while deep learning typically demands high computational power.

Diverse urban sidewalk environments used for system testing and data collection.

System Architecture

The proposed system adopts a hierarchical "Primitive" and "Advanced" perception approach, functioning similarly to the Advanced Driver Assistance Systems (ADAS) found in autonomous vehicles.

Primitive Perception

This layer serves as an "always-on" safety mechanism. Utilizing the stereo depth sensors of an OAK-D camera, the system generates a point cloud of the user's immediate surroundings. By dividing the field of view into a grid and monitoring obstacle density within 1.5 meters, the system can quickly identify potential collisions and estimate obstacle shapes by fitting cuboids to the detected points.

Advanced Perception

The advanced layer provides semantic understanding of the scene. It employs SSD-MobileNet for detecting objects such as cars, pedestrians, and signs. Furthermore, specialized models handle semantic segmentation to distinguish between roads, sidewalks, and crosswalks.

Integrated perception pipeline showing object detection, depth analysis, and spatial mapping.

Edge Optimization and UI

To maintain high performance on low-power hardware, models are optimized using OpenVINO and TensorFlow Lite. Much of the processing is offloaded to the OAK-D’s on-chip Intel Myriad X AI processor. The user interface utilizes a voice-based system (Vosk for speech-to-text and GoogleTTS for feedback) operating on an asynchronous stack to ensure that verbal descriptions do not interrupt real-time obstacle detection.

Data and Training

The system's robust performance is underpinned by a combination of public benchmarks (Google Open Image, Cityscapes, LISA) and custom-collected data. The researchers gathered approximately 9,000 images from Monrovia, CA, to create a specialized Curb Dataset for elevation change detection, alongside a local Traffic Sign Dataset.

Key Results and Insights

  • Performance: The system achieves a real-time inference rate of 22 FPS, sufficient for pedestrian speeds.
  • Accuracy: The traffic sign classifier reached F1-scores between 0.98 and 1.00. A novel MiniVGGNet ensemble approach for curb detection achieved 96-97% accuracy, overcoming the limitations of low-resolution depth sensors.
  • Spatial Awareness: The system uses "clock-face" notation (e.g., "Car at 10 o'clock") to provide users with precise spatial mapping.
  • Portability: The entire system, powered by a 10,000 mAh battery, offers 8 hours of operation and fits within a standard vest and small backpack.

Conclusion and Future Work

While the system successfully integrates complex perception tasks on the edge, full-scene semantic segmentation remains computationally expensive, currently limited to ~1.5 FPS. Future iterations aim to replace the host laptop with a mobile device or dedicated edge board like the Nvidia Jetson to further improve portability and discretion. Expansion to detect more complex structures, such as staircases, is also planned.

Summary updated

BibTeX
@inproceedings{mahendran2021computer,
  author = {Jagadish K. Mahendran and Daniel T. Barry and Anita K. Nivedha and Suchendra M. Bhandarkar},
  title = {{Computer Vision-Based Assistance System for the Visually Impaired Using Mobile Edge Artificial Intelligence}},
  booktitle = {Workshops @ CVPR},
  year = {2021},
  doi = {10.1109/cvprw53098.2021.00274},
}
Figure