Abstract: Although cluttered indoor scenes contain significant high-level semantic information useful for mapping and localization, most Visual Odometry (VO) algorithms rely primarily on geometric features like points, lines, and planes. While joint optimization of semantic labels and odometry offers accuracy, it is often computationally slow. Conversely, direct sparse approaches for VO have achieved a balance between speed and accuracy. We present SalientDSO, a framework that incorporates semantic information via visual saliency into Direct Sparse Odometry (DSO). By leveraging deep learning for visual saliency and scene parsing, SalientDSO drives feature selection to achieve highly accurate and robust VO, even with as few as 40 point features per frame. Evaluations on the ICL-NUIM, TUM monoVO, and our new CVL-UMD datasets demonstrate that SalientDSO outperforms state-of-the-art methods like DSO and ORB-SLAM.
Introduction
Visual Odometry (VO) is a fundamental component for autonomous navigation, estimating camera motion through image sequences. Traditional "direct" methods, such as DSO, identify features based on low-level intensity gradients. However, in cluttered indoor environments, many high-gradient areas—such as shadows or repetitive floor patterns—are poor landmarks for long-term tracking.
SalientDSO addresses this by mimicking human visual attention. Instead of treating all gradients equally, the system focuses on "salient" objects that provide stable, informative landmarks, effectively bridging the gap between low-level geometry and high-level semantic understanding.
The SalientDSO Framework
SalientDSO introduces a semantic pre-processing layer to the standard DSO backbone. The system architecture integrates deep learning models to guide the selection of candidate points for tracking and optimization.
Methodology
- Saliency Prediction: The system uses SalGAN, a Generative Adversarial Network, to generate a saliency map estimating human fixations within the scene.
- Semantic Filtering: To refine the often "fuzzy" saliency maps, PSPNet (Pyramid Scene Parsing Network) provides semantic labels. The system down-weights saliency in non-informative structural regions like floors and ceilings.
- Feature Selection: The filtered saliency map acts as a probability distribution. SalientDSO samples patches from high-saliency areas and then selects pixels with high gradients within those patches. This ensures points are both semantically meaningful and geometrically distinct.
- Optimization: The core tracking and bundle adjustment are handled by the original DSO framework, maintaining high-speed photometric refinement.
Experimental Results
The researchers evaluated SalientDSO across three benchmarks: the synthetic ICL-NUIM, the real-world TUM monoVO, and the newly released CVL-UMD dataset, which features cluttered indoor sequences captured with a Parrot SLAMDunk sensor.
Key Findings
- Superior Accuracy: SalientDSO consistently outperformed standard DSO and ORB-SLAM on the ICL-NUIM dataset in terms of Root Mean Square Error (RMSE).
- Extreme Robustness: A critical advantage of SalientDSO is its performance at low point densities. In stress tests using only 40 points per frame, SalientDSO maintained a 92-96% success rate on the CVL dataset, while standard DSO's success rate plummeted to 53-59% due to scale drift and tracking failures.
- Reduced Drift: Qualitative analysis of generated point clouds shows that SalientDSO produces better alignment of repetitive structures, indicating significantly reduced rotational and translational drift compared to baseline methods.
Conclusion
SalientDSO demonstrates that "active" vision—focusing on semantically relevant landmarks—significantly enhances the robustness and accuracy of direct sparse odometry. By filtering out distractor textures and prioritizing salient objects, the system achieves stable tracking even under extreme sparsity. Future work will focus on optimizing the deep learning components for real-time performance on mobile hardware and extending the semantic weighting logic to outdoor environments.
Summary updated