Helping the Blind to Get through COVID-19: Social Distancing Assistant Using Real-Time Semantic Segmentation on RGB-D Video

Manuel Martínez, Kailun Yang, Angela Constantinescu, Rainer Stiefelhagen

TL;DR

This paper presents a wearable system that uses an RGB-D camera and real-time semantic segmentation to help visually impaired individuals maintain social distancing during the COVID-19 pandemic. The system provides audio feedback through bone-conducting headphones when a person is detected within a critical proximity, offering a crucial assistive tool for navigating public spaces.

What Problem Does This Paper Address?

The COVID-19 pandemic introduced social distancing as a critical public health measure. However, for blind and visually impaired individuals, adhering to these guidelines presents significant challenges. They often struggle to perceive the distance to others, cannot see visual cues like floor markings, and existing assistive technologies are not designed to address this specific need. This can lead to stressful situations and social isolation, forcing many to avoid public spaces altogether. This paper aims to bridge this gap by providing a technological solution that enhances situational awareness for social distancing.

Methods

The core of the system relies on a combination of computer vision techniques and specialized hardware.

  • Perception Hardware: The system utilizes KR-Vision glasses equipped with an Intel RealSense RGB-D camera. This camera captures both color (RGB) and depth information simultaneously. Bone-conducting headphones are integrated into the glasses for audio feedback.
  • Real-Time Semantic Segmentation: A deep learning-based semantic segmentation algorithm, specifically SwaftNet, is employed on the RGB camera feed. This algorithm is trained to identify and delineate various objects in the scene, with a particular focus on detecting "person" class pixels. The use of semantic segmentation, as opposed to simpler object detection, allows for more precise localization of individuals.
  • Depth Integration: The depth information from the RGB-D camera is then fused with the semantic segmentation output. This allows the system to calculate the precise distance to the detected persons.
  • Audio Feedback: If a person is detected within a predefined range (defaulting to 0.5m to 1.5m), the system generates an audio alert. This feedback is delivered via bone-conducting headphones, which transmit sound through the skull, allowing the user to simultaneously hear ambient sounds. The sonification is designed to be non-intrusive, with volume and pitch modulated to indicate the confidence and urgency of the detection.
  • System Integration: The entire system is orchestrated using the Robotic Operating System (ROS), facilitating seamless communication between different software components written in various languages (C++ and Python).

Datasets

  • Training: The SwaftNet model was trained on the Mapillary Vistas dataset. This dataset is described as being designed for street-level context and includes a "person" class, making it suitable for the task.
  • Evaluation: For evaluating the real-world performance of the trained model, the authors used the PASS dataset. This dataset was captured using a wearable navigation assistance system (head-mounted lenses), making it a more relevant benchmark for the egocentric perspective of their system compared to Mapillary Vistas.

Key Insights and Ideas

  • Bridging the Gap for Visually Impaired: The paper highlights a critical, unmet need for assistive technology in the context of social distancing, demonstrating how computer vision can directly address public health challenges for vulnerable populations.
  • Semantic Segmentation for Precision: The choice of semantic segmentation over bounding-box-based object detection is crucial. It allows for more accurate identification of person pixels, leading to more precise distance estimations and thus more reliable alerts.
  • Non-Intrusive Audio Feedback: The use of bone-conducting headphones and a sonification strategy that only alerts when necessary (i.e., when a person is too close) is key to user acceptance and integration with other assistive devices. The system avoids constant, potentially annoying, alerts.
  • Low Cognitive Load: The system's design aims to minimize cognitive burden on the user, allowing them to focus on navigation and interaction rather than constantly processing system feedback.

Key Results

  • User Study: A qualitative user study with one blind and four blindfolded participants found the system to be precise, easy to use, and associated with a low cognitive load (average RTLX score of 3 out of 20). Participants generally preferred using the system over not using it.
  • Usability: The System Usability Scale (SUS) scores were high, ranging from 77.5% to 95.0% (mean of 86.5), indicating good perceived usability. The blind participant achieved the highest SUS score.
  • Performance: The system achieved a mean IoU of 50.8% for person segmentation on the PASS dataset at a resolution of 320x240, with a delay of approximately 107.9 ms per frame on the laptop and 46.7 ms on the Nvidia Xavier platform. This allowed for nearly 10 frames per second processing, providing timely feedback.
  • Qualitative Feedback: Blind users expressed enthusiasm for the system, highlighting its potential to significantly improve their ability to navigate public spaces safely and comfortably, not just during the pandemic but also as a general assistive tool.

Limitations and Future Work

  • Hardware Integration: The current prototype uses a laptop as a processing platform, which is not ideal for a wearable system. The authors acknowledge the need for integration into a smaller package.
  • Processing Platform: While the system performed well on a laptop and showed promise on the Nvidia Xavier, further exploration of even smaller embedded hardware solutions (e.g., TensorFlow Lite, Coral, Movidius) is planned.
  • Audio Feedback Intuitiveness: While understood, the frequency mapping for urgency was deemed not entirely intuitive by users.
  • Excessive Heat and Delay: During initial testing, the laptop experienced throttling due to heat and a software bug caused significant delays. These issues were addressed, but the need for better ventilation in wearable systems remains a consideration.
  • Generalization: While the system shows good generalization to unseen domains, including indoor scenes, further validation in diverse real-world scenarios is always beneficial.

Summary updated

Abstract

The current COVID-19 pandemic is having a major impact on our daily lives. Social distancing is one of the measures that has been implemented with the aim of slowing the spread of the disease, but it is difficult for blind people to comply with this. In this paper, we present a system that helps blind people to maintain physical distance to other persons using a combination of RGB and depth cameras. We use a real-time semantic segmentation algorithm on the RGB camera to detect where persons are and use the depth camera to assess the distance to them; then, we provide audio feedback through bone-conducting headphones if a person is closer than 1.5 m. Our system warns the user only if persons are nearby but does not react to non-person objects such as walls, trees or doors; thus, it is not intrusive, and it is possible to use it in combination with other assistive devices. We have tested our prototype system on one blind and four blindfolded persons, and found that the system is precise, easy to use, and amounts to low cognitive load.
BibTeX
@inproceedings{martinez2020helping,
  author = {Manuel Martínez and Kailun Yang and Angela Constantinescu and Rainer Stiefelhagen},
  title = {{Helping the Blind to Get through COVID-19: Social Distancing Assistant Using Real-Time Semantic Segmentation on RGB-D Video}},
  booktitle = {Sensors},
  year = {2020},
  url = {https://openreview.net/forum?id=q0qnZ-ZRmY},
}
Figure