SalientDSO: Bringing Attention to Direct Sparse Odometry

Huai-Jen Liang, Nitin J. Sanket, Cornelia Fermüller, Yiannis Aloimonos

Abstract: Although cluttered indoor scenes contain significant high-level semantic information useful for mapping and localization, most Visual Odometry (VO) algorithms rely primarily on geometric features like points, lines, and planes. While joint optimization of semantic labels and odometry offers accuracy, it is often computationally slow. Conversely, direct sparse approaches for VO have achieved a balance between speed and accuracy. We present SalientDSO, a framework that incorporates semantic information via visual saliency into Direct Sparse Odometry (DSO). By leveraging deep learning for visual saliency and scene parsing, SalientDSO drives feature selection to achieve highly accurate and robust VO, even with as few as 40 point features per frame. Evaluations on the ICL-NUIM, TUM monoVO, and our new CVL-UMD datasets demonstrate that SalientDSO outperforms state-of-the-art methods like DSO and ORB-SLAM.

Introduction

Visual Odometry (VO) is a fundamental component for autonomous navigation, estimating camera motion through image sequences. Traditional "direct" methods, such as DSO, identify features based on low-level intensity gradients. However, in cluttered indoor environments, many high-gradient areas—such as shadows or repetitive floor patterns—are poor landmarks for long-term tracking.

SalientDSO addresses this by mimicking human visual attention. Instead of treating all gradients equally, the system focuses on "salient" objects that provide stable, informative landmarks, effectively bridging the gap between low-level geometry and high-level semantic understanding.

The SalientDSO Framework

SalientDSO introduces a semantic pre-processing layer to the standard DSO backbone. The system architecture integrates deep learning models to guide the selection of candidate points for tracking and optimization.

The SalientDSO system architecture, integrating scene parsing and saliency prediction into the standard DSO tracking and optimization pipeline.

Methodology

  1. Saliency Prediction: The system uses SalGAN, a Generative Adversarial Network, to generate a saliency map estimating human fixations within the scene.
  2. Semantic Filtering: To refine the often "fuzzy" saliency maps, PSPNet (Pyramid Scene Parsing Network) provides semantic labels. The system down-weights saliency in non-informative structural regions like floors and ceilings.
  3. Feature Selection: The filtered saliency map acts as a probability distribution. SalientDSO samples patches from high-saliency areas and then selects pixels with high gradients within those patches. This ensures points are both semantically meaningful and geometrically distinct.
  4. Optimization: The core tracking and bundle adjustment are handled by the original DSO framework, maintaining high-speed photometric refinement.
Comparison of feature selection strategies. The bottom row shows the input image, predicted saliency map, and semantic segmentation used to guide point selection.

Experimental Results

The researchers evaluated SalientDSO across three benchmarks: the synthetic ICL-NUIM, the real-world TUM monoVO, and the newly released CVL-UMD dataset, which features cluttered indoor sequences captured with a Parrot SLAMDunk sensor.

Key Findings

  • Superior Accuracy: SalientDSO consistently outperformed standard DSO and ORB-SLAM on the ICL-NUIM dataset in terms of Root Mean Square Error (RMSE).
  • Extreme Robustness: A critical advantage of SalientDSO is its performance at low point densities. In stress tests using only 40 points per frame, SalientDSO maintained a 92-96% success rate on the CVL dataset, while standard DSO's success rate plummeted to 53-59% due to scale drift and tracking failures.
  • Reduced Drift: Qualitative analysis of generated point clouds shows that SalientDSO produces better alignment of repetitive structures, indicating significantly reduced rotational and translational drift compared to baseline methods.

Conclusion

SalientDSO demonstrates that "active" vision—focusing on semantically relevant landmarks—significantly enhances the robustness and accuracy of direct sparse odometry. By filtering out distractor textures and prioritizing salient objects, the system achieves stable tracking even under extreme sparsity. Future work will focus on optimizing the deep learning components for real-time performance on mobile hardware and extending the semantic weighting logic to outdoor environments.

Summary updated

Abstract

Although cluttered indoor scenes have a lot of useful high-level semantic information which can be used for mapping and localization, most Visual Odometry (VO) algorithms rely on the usage of geometric features such as points, lines and planes. Lately, driven by this idea, the joint optimization of semantic labels and obtaining odometry has gained popularity in the robotics community. The joint optimization is good for accurate results but is generally very slow. At the same time, in the vision community, direct and sparse approaches for VO have stricken the right balance between speed and accuracy. We merge the successes of these two communities and present a way to incorporate semantic information in the form of visual saliency to Direct Sparse Odometry - a highly successful direct sparse VO algorithm. We also present a framework to filter the visual saliency based on scene parsing. Our framework, SalientDSO, relies on the widely successful deep learning based approaches for visual saliency and scene parsing which drives the feature selection for obtaining highly-accurate and robust VO even in the presence of as few as 40 point features per frame. We provide extensive quantitative evaluation of SalientDSO on the ICL-NUIM and TUM monoVO datasets and show that we outperform DSO and ORB-SLAM - two very popular state-of-the-art approaches in the literature. We also collect and publicly release a CVL-UMD dataset which contains two indoor cluttered sequences on which we show qualitative evaluations. To our knowledge this is the first paper to use visual saliency and scene parsing to drive the feature selection in direct VO.
BibTeX
@article{liang2019salientdso,
  author = {Huai-Jen Liang and Nitin J. Sanket and Cornelia Fermüller and Yiannis Aloimonos},
  title = {{SalientDSO: Bringing Attention to Direct Sparse Odometry}},
  journal = {IEEE Trans Autom. Sci. Eng.},
  year = {2019},
  doi = {10.1109/tase.2019.2900980},
}
Figure