Expanding the Detection of Traversable Area with RealSense for the Visually Impaired

Kailun Yang, Kaiwei Wang, Weijian Hu, Jian Bai

TL;DR

This paper presents a novel approach to significantly expand the detection range of traversable areas for visually impaired individuals using the Intel RealSense R200 RGB-D sensor. By enhancing depth images with IR image matching and RGB-guided filtering, and then employing a seeded region growing algorithm, the system achieves a much wider and more reliable traversable area awareness, crucial for safe navigation.

What Problem Does This Paper Address?

Visually impaired individuals (VIP) face significant challenges navigating the world. While RGB-Depth (RGB-D) sensors offer promising assistive technology, their inherent limitations—specifically a narrow field of view and sparse depth data at longer distances—severely restrict their ability to provide comprehensive traversable area awareness. This limitation hinders effective path planning and obstacle avoidance, especially in complex indoor and outdoor environments. Existing solutions often struggle with accuracy, range, or robustness in varying lighting conditions, leaving a critical gap in providing reliable, long-range navigation assistance.

Methods

The proposed approach tackles the limitations of standard RGB-D sensors through a multi-stage process:

  1. Depth Image Enhancement:
    • IR Image Large-Scale Matching: The original sparse depth map from the RealSense R200 is augmented by performing large-scale stereo matching on the captured IR images. This process leverages textures to generate a denser depth map.
  2. RGB Image-Guided Filtering: To further refine the depth map, a guided filter is applied, incorporating color information from the RGB image. This helps to fill holes, reduce noise, and estimate depth in unknown areas by assuming that pixels with similar colors likely share similar depth values. This step also effectively extends the depth perception to the wider field of view of the RGB camera.
  3. Preliminary Ground Segmentation:
    • 3D Point Cloud Generation and World Coordinate Transformation: The enhanced depth data is converted into a 3D point cloud. An attitude sensor (MPU6050) is used to transform these points into a world coordinate system, correcting for the sensor's orientation and reducing sample errors.
  4. RANSAC for Plane Detection: The Random Sample Consensus (RANSAC) algorithm is employed to robustly identify the dominant ground plane from the point cloud. This initial segmentation is constrained by the sensor's attitude and real-world height estimations.
  5. Surface Normal Vector Estimation: Salient parts and small obstacles on the detected ground plane are removed by estimating surface normal vectors. Patches with normal vectors not aligned with the vertical direction are discarded, ensuring only true traversable surfaces are considered.
  6. Seeded Region Growing for Expansion:
    • Seed Selection: Seeds for region growing are chosen from the preliminarily segmented ground plane, ensuring they are not on the edges of the color image and have valid neighbors.
  7. Growth Conditions: The algorithm expands the traversable area by considering four conditions for neighboring pixels: not being on Canny edges, not having been traversed, having a reasonable real height, and having a small difference in depth and hue compared to the current pixel. This combines depth and color information to guide the expansion.
  8. Stable Expansion: Unlike methods that rely on random seed selection, this approach uses pre-segmented pixels, leading to more stable and consistent traversable area detection across frames.

Datasets

The paper does not explicitly mention training datasets. The approach is evaluated using data captured from the Intel RealSense R200 sensor in various indoor and outdoor scenarios, including offices, corridors, roads, playgrounds, and school roads. The authors also conducted a user study with eight visually impaired volunteers, which serves as a form of real-world evaluation.

Key Insights and Ideas

  • Leveraging RGB for Depth Enhancement: The core insight is that combining RGB information with depth data, even when the depth data is sparse or noisy at range, can significantly improve the quality and density of the depth map. The RGB-guided filter is a powerful tool for this.
  • Expanding the Field of View: By effectively using the wider field of view of the RGB camera to guide depth estimation, the system overcomes the narrow depth field angle limitation of the RealSense's IR cameras, leading to broader traversable area detection.
  • Robustness through Multi-Sensor Fusion: Integrating an attitude sensor (MPU6050) with the RGB-D sensor provides crucial information for transforming point clouds into a world coordinate system, significantly reducing errors and improving the robustness of the initial ground plane detection.
  • Stable and Intelligent Region Growing: The seeded region growing approach, starting from robustly detected ground planes and considering multiple cues (depth, color, edges), provides a more stable and intelligent way to expand traversable areas compared to purely depth-based or random seed methods.
  • Practical Usability for VIP: The integration of the system into a wearable prototype with a non-semantic stereophonic audio interface demonstrates a commitment to real-world application and user experience for visually impaired individuals.

Key Results

  • Expanded Traversable Area: The system successfully expands the traversable area detection range significantly, increasing the horizontal field angle from the sensor's native 59° to approximately 70° (matching the RGB camera's field of view).
  • Improved Depth Map Density: The IR image large-scale matching and RGB image guided filtering process results in depth maps with near 100% density, a substantial improvement over the original sensor output (around 60-80% density).
  • High Detection Rate: The approach achieves a detection rate of over 90% across various scenarios.
  • Low Expansion Error: The average expansion error (classifying non-traversable areas as traversable) is reported as 7.8%, demonstrating the reliability of the system in avoiding hazardous obstacles.
  • User Study Performance: In a user study with eight visually impaired volunteers, the system using expanded traversable area detection resulted in:
  • 78.6% fewer collisions.
  • 29.5% less time to complete a course.
  • 43.4% fewer steps to complete a course.
  • Processing Speed: The average processing time per frame is 610 ms on a 1.90 GHz Intel Core 5 processor, enabling a VIP audio feedback refresh rate of 1.6 times per second.

Limitations and Future Work

  • Processing Speed Optimization: The authors acknowledge that the algorithm's implementation is not yet fully optimized and express a desire to speed it up for even faster processing.
  • Cross-Modal Stereo Matching: Future work includes exploring cross-modal stereo matching between IR and RGB images to further improve detection range and accuracy.
  • Expansion Error in Specific Scenarios: While generally robust, an example of expansion error is shown where a part of a car was unexpectedly classified as traversable area, indicating that challenging scenarios can still lead to minor inaccuracies.
  • User Feedback: Some users provided advice for future enhancements, such as adding face recognition or GPS navigation, and suggesting the prototype be designed as a hat.

Summary updated

Abstract

The introduction of RGB-Depth (RGB-D) sensors into the visually impaired people (VIP)-assisting area has stirred great interest of many researchers. However, the detection range of RGB-D sensors is limited by narrow depth field angle and sparse depth map in the distance, which hampers broader and longer traversability awareness. This paper proposes an effective approach to expand the detection of traversable area based on a RGB-D sensor, the Intel RealSense R200, which is compatible with both indoor and outdoor environments. The depth image of RealSense is enhanced with IR image large-scale matching and RGB image-guided filtering. Traversable area is obtained with RANdom SAmple Consensus (RANSAC) segmentation and surface normal vector estimation, preliminarily. A seeded growing region algorithm, combining the depth image and RGB image, enlarges the preliminary traversable area greatly. This is critical not only for avoiding close obstacles, but also for allowing superior path planning on navigation. The proposed approach has been tested on a score of indoor and outdoor scenarios. Moreover, the approach has been integrated into an assistance system, which consists of a wearable prototype and an audio interface. Furthermore, the presented approach has been proved to be useful and reliable by a field test with eight visually impaired volunteers.
BibTeX
@inproceedings{yang2016expanding,
  author = {Kailun Yang and Kaiwei Wang and Weijian Hu and Jian Bai},
  title = {{Expanding the Detection of Traversable Area with RealSense for the Visually Impaired}},
  booktitle = {Sensors},
  year = {2016},
  url = {https://openreview.net/forum?id=Y5HXKlobaL9q},
}
Figure