TL;DR
This paper introduces a unified approach to real-time semantic segmentation for visually impaired individuals, enabling them to perceive their environment more comprehensively. By leveraging a novel deep learning architecture, the system effectively integrates various navigation-critical tasks like traversability assessment and obstacle avoidance into a single, efficient pipeline, significantly enhancing safety and independence.
What Problem Does This Paper Address?
Visually impaired individuals rely heavily on assistive technologies for safe and independent navigation. Existing solutions often employ separate detectors for different environmental features (e.g., traversable areas, obstacles, stairs, water hazards). While effective individually, running these multiple detectors concurrently leads to increased latency and computational burden, making them impractical for real-time assistance during dynamic navigation. This paper addresses the challenge of unifying these diverse perception needs into a single, efficient system that can provide comprehensive terrain awareness and obstacle avoidance in real-time.
Methods
The core of the proposed solution is a real-time pixel-wise semantic segmentation network. This network is built upon an encoder-decoder architecture, inspired by existing models like SegNet and ENet, but with key modifications for efficiency and effectiveness in this specific application.
- Unified Perception: Instead of separate detectors, the system uses semantic segmentation to classify every pixel into categories relevant for navigation (e.g., traversable area, sidewalk, stairs, water hazard, pedestrian, vehicle). This unified approach allows for a richer understanding of the scene.
- Efficient Architecture: The architecture features a custom "Non-bottleneck-1D" (Non-bt-1D) residual block, which leverages 1D filter factorizations to reduce computational cost and parameters while maintaining learning capacity. This is crucial for real-time performance on wearable devices.
- Depth Integration: The system integrates robust depth segmentation, obtained from a wearable RGB-D sensor. This depth information is pre-processed to create a dense depth map, which is then used in conjunction with the semantic segmentation output for obstacle avoidance.
- Pyramid Pooling Decoder: The decoder incorporates a pyramid pooling module (similar to PSPNet) to capture multi-scale contextual information, improving the accuracy of scene understanding, especially for distinguishing between similar-looking but functionally different areas (e.g., sidewalks vs. roads).
- Stereo Sound Feedback: The system provides real-time acoustic feedback through bone-conduction headphones. This feedback is sonified based on the semantic segmentation results, conveying information about traversable paths, obstacle avoidance, and specific hazards like stairs.
Datasets
- Training: The system was trained on a combination of large-scale datasets:
- ADE20K: A challenging dataset for scene parsing, including indoor and outdoor scenarios with traversability-related classes.
- PASCAL-Context: Used to enrich the training data with specific classes like sky, floor, road, grass, sidewalk, ground, water, and stairs.
- COCO-Stuff 10K: Another dataset contributing to the training data.
- A total of 37,075 images were used for training, with 24 merged classes (e.g., water hazards, stairs, and 22 frequent scene elements).
- Validation: 2000 images from ADE20K were used for validation.
- Real-World Datasets:
- TerrainAwarenessDataset [71]: A newly introduced egocentric dataset captured with the assistive prototype in real-world scenarios, used for evaluation.
- Gardens Point dataset [98]: Used for analyzing indoor/outdoor and day/night detection performance.
Key Insights and Ideas
- Unification is Key: The paper strongly advocates for unifying diverse perception tasks (traversability, obstacle avoidance, etc.) into a single semantic segmentation framework. This is presented as a more computationally efficient and effective approach than using multiple specialized detectors.
- Real-time Performance is Paramount: For visually impaired users, real-time feedback is not just desirable but critical for safety. The paper emphasizes achieving high frame rates (e.g., 62.5 FPS) on accessible hardware.
- Context Matters More Than Pixel-Exactness: For visually impaired users, understanding the general traversable path and identifying major hazards is more important than pixel-perfect segmentation of fine details. The architecture's design, particularly the decoder, prioritizes contextual information.
- Stereo Sound is an Effective Feedback Modality: The use of stereo sound, mapped to semantic information, provides intuitive and effective guidance for navigation, complementing the visual information processed by the system. The distinct sounds for different hazards (e.g., clarinet for obstacle avoidance, water droplet for traversability awareness) are crucial for avoiding confusion.
- Wearable Design is Crucial for Adoption: The system is integrated into a pair of smart glasses, a form factor that is generally well-received and less stigmatizing than other assistive devices.
Key Results
- Real-Time Performance: The proposed ERF-PSPNet achieves a total computation time of 16 ms per frame (13 ms for segmentation), enabling a refresh rate of 62.5 FPS on a single GTX1050Ti GPU. On an embedded Tegra TX1, it achieves approximately 22.0 FPS.
- Accuracy:
- On the ADE20K dataset, the proposed ERF-PSPNet achieved a Mean IoU of 66.0%, outperforming other state-of-the-art real-time segmentation networks like ENet (58.4%) and ERFNet (63.1%).
- On the real-world TerrainAwarenessDataset, the system achieved a traversable area IoU of 82.1% and a pixel-wise accuracy of 93.1%, demonstrating strong performance in diverse real-world conditions.
- Qualitative Results: Visualizations show robust segmentation of traversable areas, sidewalks, stairs, and obstacles across various scenarios, including challenging lighting and indoor/outdoor environments.
- Field Test: A closed-loop field test with six visually-impaired users demonstrated the system's effectiveness and versatility. Participants successfully navigated a ~85m course with minimal collisions, indicating the system's ability to support safe and independent navigation.
Limitations and Future Work
- Limitations:
- The paper acknowledges that the "Non-bottleneck-1D" design, while efficient, might have slight trade-offs in accuracy compared to more complex architectures.
- In some challenging real-world scenarios, the system occasionally failed to detect low-lying obstacles (e.g., wood) or had issues with sparse depth information (e.g., near fences), leading to collisions.
- The current system's robustness in varying environmental conditions (e.g., extreme weather) could be further improved.
- Future Work:
- Pixel-wise polarization estimation and multi-modal sensory awareness to enhance robustness across seasons.
- Deep learning-based depth interpolation to improve RGB-D perception in high dynamic environments and expand detectable range.
- Incorporating detection of intersection-centered scene elements like zebra crosswalks and traffic lights.
- Developing panoramic semantic segmentation for superior 360° awareness.
- Conducting larger studies with visually-impaired participants and comparing different sonification methods.
- Exploring semantics-aware SLAM by leveraging indoor semantic masks.
Summary updated