Indoor Navigation Assistance for Visually Impaired People via Dynamic SLAM and Panoptic Segmentation with an RGB-D Sensor

Wenyan Ou, Jiaming Zhang, Kunyu Peng, Kailun Yang, Gerhard Jaworek, Karin Müller, Rainer Stiefelhagen

Abstract

Exploring an unfamiliar indoor environment and avoiding obstacles is challenging for visually impaired people. Currently, several approaches achieve the avoidance of static obstacles based on the mapping of indoor scenes. To solve the issue of distinguishing dynamic obstacles, we propose an assistive system with an RGB-D sensor to detect dynamic information of a scene. Once the system captures an image, panoptic segmentation is performed to obtain the prior dynamic object information. With sparse feature points extracted from images and the depth information, poses of the user can be estimated. After the ego-motion estimation, the dynamic object can be identified and tracked. Then, poses and speed of tracked dynamic objects can be estimated, which are passed to the users through acoustic feedback.

Introduction

Navigating complex indoor environments is a significant challenge for visually impaired individuals. While existing navigation aids effectively map static obstacles like walls and furniture, they often struggle in dynamic settings—such as busy hallways or malls—where people and objects are in motion. Traditional Simultaneous Localization and Mapping (SLAM) algorithms typically treat moving objects as "noise" to be filtered out to maintain camera localization. However, for a visually impaired user, these moving objects represent high-risk obstacles that must be tracked. This research presents a wearable system designed to distinguish between the user's ego-motion and the movement of dynamic objects, providing real-time acoustic feedback for safer navigation.

Methodology

The proposed system utilizes a wearable hardware suite consisting of smart glasses with an RGB-D sensor, a backpack-mounted laptop for processing, and bone-conduction headphones for user feedback.

System hardware and software pipeline overview.

The software architecture follows a multi-stage pipeline:

  1. Preprocessing: Every frame undergoes panoptic segmentation using PanopticFCN to identify "prior dynamic" classes (e.g., people) and generate instance masks. Human joint estimation via OpenPose can further stabilize motion tracking.
  2. Ego-Motion Estimation: The system calculates the user's position and orientation using sparse feature points from static environmental elements.
  3. Dynamic Object Identification: Objects are categorized as "Prior Dynamic" (known moving classes verified via 3D scene flow) or "Non-Prior Dynamic" (static classes that are currently moving, detected via depth differences).
  4. Tracking and Feedback: Moving objects are tracked using DIS optical flow to estimate linear velocity and distance. This information is converted into verbal cues, such as "One moving person, distance 1.8m, direction directly forward, low speed."

Key Insights and Innovations

  • Semantic-Aware SLAM: By integrating panoptic segmentation, the system understands the semantic nature of objects, allowing it to differentiate between reliable landmarks for localization and potential hazards.
  • Mask-Based Filtering: Instead of computationally expensive region-growing techniques, the system uses instance masks to efficiently calculate the density of dynamic points within an object.
  • Stable Human Tracking: To improve speed estimation, the system focuses on "stable" human joints (shoulders and hips) which deform less than extremities during movement, leading to more accurate trajectory data.

Experimental Results

The system was evaluated using the TUM and Bonn RGB-D datasets, as well as real-world trials.

Visual processing steps: (a) RGB frame, (b) panoptic segmentation, (c) dynamic object identification, (d) speed estimation, and (e) trajectory mapping.

Key findings include:

  • Enhanced Accuracy: In highly dynamic scenarios, the system significantly outperformed standard ORB-SLAM2. In tests involving obstructing moving boxes, the Absolute Trajectory Error (ATE) was reduced from 0.471m to 0.196m.
  • Real-Time Performance: The system maintains a processing speed of 4–7 FPS on mobile hardware, which is sufficient for safe walking-speed navigation.
  • Robustness: The system maintains localization even when large portions of the field of view are obscured by moving objects, a common failure point for traditional SLAM.

Conclusion and Future Work

While the system demonstrates a robust foundation for dynamic indoor navigation, several areas for improvement remain. Future work will focus on reducing dependency on segmentation accuracy, optimizing the computational load for more crowded environments, and conducting user studies with visually impaired volunteers to refine the acoustic interface for better intuitiveness.

Summary updated

Abstract

Exploring an unfamiliar indoor environment and avoiding obstacles is challenging for visually impaired people. Currently, several approaches achieve the avoidance of static obstacles based on the mapping of indoor scenes. To solve the issue of distinguishing dynamic obstacles, we propose an assistive system with an RGB-D sensor to detect dynamic information of a scene. Once the system captures an image, panoptic segmentation is performed to obtain the prior dynamic object information. With sparse feature points extracted from images and the depth information, poses of the user can be estimated. After the ego-motion estimation, the dynamic object can be identified and tracked. Then, poses and speed of tracked dynamic objects can be estimated, which are passed to the users through acoustic feedback.
BibTeX
@inproceedings{ou2022indoor,
  author = {Wenyan Ou and Jiaming Zhang and Kunyu Peng and Kailun Yang and Gerhard Jaworek and Karin Müller and Rainer Stiefelhagen},
  title = {{Indoor Navigation Assistance for Visually Impaired People via Dynamic SLAM and Panoptic Segmentation with an RGB-D Sensor}},
  booktitle = {ICCHP-AAATE},
  year = {2022},
  doi = {10.1007/978-3-031-08648-9\_19},
}
Figure