Indoor Navigation Assistance for Visually Impaired People via Dynamic SLAM and Panoptic Segmentation with an RGB-D Sensor

Wenyan Ou, Jiaming Zhang, Kunyu Peng, Kailun Yang, Gerhard Jaworek, Karin Müller, Rainer Stiefelhagen

TL;DR

Researchers have developed a wearable assistive system that helps visually impaired individuals navigate complex indoor environments by identifying and tracking moving obstacles in real-time. By combining Panoptic Segmentation (which identifies what and where objects are) with Dynamic SLAM (Simultaneous Localization and Mapping), the system provides users with acoustic feedback regarding the distance, direction, and speed of nearby people and moving objects.

What Problem Does This Paper Address?

Most current indoor navigation aids for the visually impaired rely on static maps. While these are great for avoiding walls and furniture, they fail in "dynamic" environments—places like busy hallways or malls where people are walking and objects are being moved.

Traditional SLAM algorithms often treat moving people as "noise" or outliers to be ignored so the camera can stay localized. However, for a visually impaired user, that "noise" is actually a high-risk obstacle. This paper tackles the challenge of distinguishing between the user's own movement (ego-motion) and the movement of objects in the scene, ensuring that dynamic obstacles are tracked rather than just filtered out.

Methods

The system consists of a pair of smart glasses equipped with an RGB-D (Red, Green, Blue + Depth) sensor, a lightweight laptop carried in a backpack for processing, and bone-conduction headphones for feedback.

The software pipeline follows these key steps:

  1. Preprocessing: The system uses PanopticFCN to perform panoptic segmentation on every frame. This identifies "prior dynamic" classes (like people or pets) and creates instance masks. It can also use OpenPose to track human joints for more stable motion estimation.
  2. Ego-Motion Estimation: Using sparse feature points from static parts of the environment, the system calculates the user's own position and orientation.
  3. Identifying Dynamic Objects: The system looks at two categories:
  4. Prior Dynamic: Objects known to move (people). The system uses 3D scene flow to verify if they are actually moving.
  5. Non-Prior Dynamic: Objects that shouldn't move but are (e.g., a box being pushed). These are detected by comparing depth differences between frames.
  6. Tracking & Feedback: Moving objects are tracked using DIS optical flow. The system calculates their linear velocity and distance, then translates this into verbal/acoustic cues (e.g., "One moving person, distance 1.8m, direction directly forward, low speed").

Datasets

The authors evaluated their system using two well-known public benchmarks:

  • TUM RGB-D Dataset: Specifically sequences with sitting and walking people to simulate office environments.
  • Bonn RGB-D Dataset: Used for more challenging scenarios, such as people moving boxes or obstructing the camera's view.
  • Real-life Scenarios: The authors also collected their own data using the wearable prototype in actual indoor settings to test trajectory estimation.

Key Insights and Ideas

  • Mask-Based Filtering: Instead of using traditional "region growing" on depth images (which is computationally expensive), the system leverages the instance masks from the panoptic segmentation to calculate the percentage of dynamic points within an object.
  • Stable Human Tracking: By focusing on "stable" human joints (shoulders and hips) rather than extremities, the system achieves more accurate speed estimation for moving people, as the trunk of the body deforms less during a stride.
  • Semantic-Aware SLAM: By knowing what an object is (a person vs. a chair), the system can make smarter decisions about which points to use for localization and which to track as potential hazards.

Key Results

  • Improved Accuracy: In highly dynamic sequences (like the "crowd" sequence in the TUM dataset), the proposed system significantly outperformed standard ORB-SLAM2. For example, in the "moving obstructing box" sequence, the system reduced the Absolute Trajectory Error (ATE) from 0.471m to 0.196m.
  • Real-Time Performance: The system achieves a processing speed of 4–7 FPS on a standard Intel i5 mobile CPU. While not as fast as some static SLAM systems, it is sufficient for walking-speed navigation.
  • Robustness: The system proved capable of maintaining localization even when a large portion of the frame was obscured by a moving object, a scenario where traditional SLAM typically fails.

Limitations and Future Work

The authors identify a few areas for improvement:

  • Segmentation Errors: The system's accuracy is heavily dependent on the panoptic segmentation. If the neural network fails to identify a person, the SLAM system might treat them as a static landmark, leading to errors.
  • Computational Load: The complexity scales with the number of dynamic objects in the room. More optimization is needed to handle very crowded environments on mobile hardware.
  • User Experience: While the technical foundation is solid, the authors plan to conduct extensive user studies with visually impaired volunteers to refine the "acoustic interface"—ensuring the audio feedback is intuitive and not overwhelming.

Summary updated

Abstract

Exploring an unfamiliar indoor environment and avoiding obstacles is challenging for visually impaired people. Currently, several approaches achieve the avoidance of static obstacles based on the mapping of indoor scenes. To solve the issue of distinguishing dynamic obstacles, we propose an assistive system with an RGB-D sensor to detect dynamic information of a scene. Once the system captures an image, panoptic segmentation is performed to obtain the prior dynamic object information. With sparse feature points extracted from images and the depth information, poses of the user can be estimated. After the ego-motion estimation, the dynamic object can be identified and tracked. Then, poses and speed of tracked dynamic objects can be estimated, which are passed to the users through acoustic feedback.
BibTeX
@inproceedings{ou2022indoor,
  author = {Wenyan Ou and Jiaming Zhang and Kunyu Peng and Kailun Yang and Gerhard Jaworek and Karin Müller and Rainer Stiefelhagen},
  title = {{Indoor Navigation Assistance for Visually Impaired People via Dynamic SLAM and Panoptic Segmentation with an RGB-D Sensor}},
  booktitle = {ICCHP-AAATE},
  year = {2022},
  doi = {10.1007/978-3-031-08648-9\_19},
}

Loading related content...

Figure