Unsupervised Collaborative Learning of Keyframe Detection and Visual Odometry Towards Monocular Deep SLAM

Lu Sheng, Dan Xu, Wanli Ouyang, Xiaogang Wang

Introduction

This research introduces the first unsupervised deep learning framework that jointly optimizes keyframe selection and visual odometry (VO). By establishing a "virtuous cycle" where motion estimation and keyframe detection mutually supervise each other, the system achieves state-of-the-art results for monocular depth and pose estimation without requiring labeled ground-truth data.

The collaborative learning framework creates a virtuous cycle between visual odometry and keyframe selection.

The Challenge in Monocular SLAM

In Monocular Visual SLAM (Simultaneous Localization and Mapping), two tasks are paramount: Visual Odometry (VO), which estimates the camera's trajectory and scene structure, and Keyframe Selection, which identifies the most representative frames to build a global map.

Traditional systems like ORB-SLAM rely heavily on keyframes to maintain long-term accuracy. However, recent deep learning approaches to VO have largely ignored keyframe selection, typically training only on short sequences of consecutive frames. This limitation hinders their ability to handle large geometric changes or long-range constraints. The proposed framework addresses this by integrating these complementary tasks into a single, end-to-end unsupervised model.

Methodology

The proposed framework consists of three jointly trained sub-networks:

  1. Depth Predictor ($D_{\phi_D}$): Estimates a per-pixel depth map from a single image.
  2. Camera Motion Estimator ($C_{\phi_C}$): Predicts the 6-DoF (degrees of freedom) relative motion between two frames.
  3. Keyframe Selector ($S_{\phi_S}$): A two-stream network that determines the similarity between a target frame and a reference keyframe.
Architecture of the proposed framework, highlighting (a) the depth and motion estimators and (b) the keyframe selector using visual and geometric cues.

Collaborative Optimization

The core innovation is the Keyframe Selector. Unlike previous methods that rely solely on visual pixels, this selector utilizes both Visual Cues (RGB features) and Geometric Cues (predicted depth maps and warping residuals).

The training process employs a Collaborative Optimization Loss with a triplet loss strategy: the network is taught that a target frame should be more similar to its temporally nearest keyframe than to a "hard negative" sample from elsewhere in the sequence. Simultaneously, the VO components use these selected keyframes to enforce stronger photometric and geometric consistency over longer distances than consecutive frames allow.

Key Insights and Results

  • Mutual Benefit: Better VO provides more accurate geometric data to the keyframe selector; in turn, robust keyframes provide better "long-distance" supervision for the VO, helping regularize motion estimation during complex maneuvers like sharp turns.
  • Adaptive Density: The model learns to pick keyframes more frequently when the camera is turning (high geometric change) and less frequently when driving in a straight line, mimicking the behavior of hand-engineered traditional SLAM systems.

Performance on KITTI Benchmark

  • Depth Estimation: The model established a new state-of-the-art on the KITTI benchmark, reducing the Squared Relative Error (sq rel) from 1.257 to 1.021.
  • Pose Estimation: In terms of Absolute Trajectory Error (ATE), the model outperformed other unsupervised deep learning baselines and approached the performance of some supervised methods.
  • Visual Quality: Predicted depth maps show sharper edges and better consistency in texture-less areas compared to previous models.

Limitations and Future Work

While the model shows significant progress, a slight performance gap remains compared to some VO-specific models in certain metrics, likely due to training on shorter 3-frame snippets. Additionally, the current "cycle consistency" loss assumes a static world, which can lead to errors in scenes with many moving objects. Future directions include integrating a robust "loop closure" mechanism to move toward a full-featured Deep SLAM system.

Summary updated

Abstract

In this paper we tackle the joint learning problem of keyframe detection and visual odometry towards monocular visual SLAM systems. As an important task in visual SLAM, keyframe selection helps efficient camera relocalization and effective augmentation of visual odometry. To benefit from it, we first present a deep network design for the keyframe selection, which is able to reliably detect keyframes and localize new frames, then an end-to-end unsupervised deep framework further proposed for simultaneously learning the keyframe selection and the visual odometry tasks. As far as we know, it is the first work to jointly optimize these two complementary tasks in a single deep framework. To make the two tasks facilitate each other in the learning, a collaborative optimization loss based on both geometric and visual metrics is proposed. Extensive experiments on publicly available datasets (i.e. KITTI raw dataset and its odometry split [12]) clearly demonstrate the effectiveness of the proposed approach, and new state-ofthe-art results are established on the unsupervised depth and pose estimation from monocular video.
BibTeX
@inproceedings{sheng2019unsupervised,
  author = {Lu Sheng and Dan Xu and Wanli Ouyang and Xiaogang Wang},
  title = {{Unsupervised Collaborative Learning of Keyframe Detection and Visual Odometry Towards Monocular Deep SLAM}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2019},
  doi = {10.1109/iccv.2019.00440},
}
Figure