TL;DR
SalientDSO enhances the popular Direct Sparse Odometry (DSO) framework by using deep learning to mimic human visual attention. By prioritizing "salient" objects and filtering out uninformative regions like floors and ceilings, the system achieves superior accuracy and extreme robustness—maintaining tracking with as few as 40 points per frame.
What Problem Does This Paper Address?
Visual Odometry (VO) is the process of estimating a camera's motion by analyzing sequences of images. Most state-of-the-art "direct" methods, like DSO, rely on low-level geometric features—essentially looking for any pixel with a high intensity gradient.
However, in cluttered indoor environments, not all gradients are created equal. A high-contrast pattern on a rug or a shadow on a blank wall provides a strong gradient but offers poor "landmark" quality for long-term tracking. Humans don't navigate this way; we use "fixations" to focus on distinct objects. The authors address the gap between low-level geometric tracking and high-level semantic understanding, aiming to make VO more "object-centric" and robust to noise.
Methods
The authors introduce SalientDSO, which adds a semantic pre-processing layer to the standard DSO backbone.
- Saliency Prediction: They use SalGAN, a Generative Adversarial Network, to predict a "saliency map" that estimates where a human would likely look in a scene.
- Semantic Filtering: Saliency maps can be "fuzzy." To sharpen them, they use PSPNet (Pyramid Scene Parsing Network) to label pixels (e.g., "wall," "floor," "table"). They then down-weight the saliency of structural elements like floors and ceilings, which are often non-informative for odometry.
- Feature Selection: Instead of selecting candidate points uniformly across the image, SalientDSO treats the filtered saliency map as a probability distribution. It samples patches where saliency is high. Within those patches, it still selects pixels with high gradients, ensuring the points are both semantically meaningful and geometrically useful.
- Backbone: The core tracking and optimization (bundle adjustment) are handled by the original DSO framework, ensuring the system benefits from DSO's high-speed photometric refinement.
Datasets
The researchers evaluated their system on three main benchmarks:
- ICL-NUIM: A synthetic dataset of indoor living rooms and offices used for quantitative trajectory error (RMSE) analysis.
- TUM monoVO: A large-scale real-world dataset. Note: Because this dataset is grayscale, the authors disabled semantic filtering here, relying only on raw saliency.
- CVL-UMD Dataset: A newly introduced dataset by the authors containing two cluttered indoor sequences captured with a Parrot SLAMDunk sensor, specifically designed to test VO in complex environments.
Key Insights and Ideas
- Active vs. Passive Vision: The paper shifts from a "passive" approach (process everything with a gradient) to an "active" approach (focus on what matters). This mimics the human visual system's ability to ignore "distractor" textures.
- Semantics as a Filter: Saliency alone isn't enough because a bright light on a flat wall is "salient" but useless for tracking. By combining saliency with scene parsing, the system learns that a "cup on a table" is a better landmark than a "shadow on the floor."
- Extreme Sparsity: The most surprising insight is the system's performance at low point densities. While standard VO often fails when forced to track very few points, SalientDSO’s "smart" point selection allows it to remain stable even with a tiny fraction of the usual data.
Key Results
- Accuracy: SalientDSO outperformed both the original DSO and ORB-SLAM (a leading feature-based method) on the ICL-NUIM dataset in terms of Root Mean Square Error (RMSE).
- Robustness at $N_p = 40$: In a stress test using only 40 points per frame, SalientDSO achieved a 92-96% success rate on the CVL dataset, whereas standard DSO's success rate dropped as low as 53-59% due to tracking failures and scale drift.
- Qualitative Improvement: Point clouds generated by SalientDSO show much better alignment of repetitive structures (like checkerboard patterns) compared to DSO, indicating significantly less rotational and translational drift.
Limitations and Future Work
- Computational Overhead: While the VO backbone is real-time, the deep learning components (SalGAN and PSPNet) currently take about one second per frame on an NVIDIA Titan-Xp. This prevents the entire pipeline from being truly real-time on mobile hardware without further optimization or dedicated AI accelerators.
- Indoor Focus: The current weighting logic is tuned for indoor scenes (down-weighting floors/ceilings). The authors note that extending this to outdoor environments—where "ground" and "sky" have different saliency profiles—is a key next step.
- Hardware Integration: The authors suggest moving the saliency/parsing logic onto dedicated hardware to achieve a full 30fps pipeline.
Summary updated