Moving SLAM: Fully Unsupervised Deep Learning in Non-Rigid Scenes

Dan Xu, Andrea Vedaldi, João F. Henriques

Abstract

This paper introduces "Moving SLAM," a framework that learns to decompose videos into 3D geometry, camera motion, and moving objects without any human supervision. By replacing the standard "rigid-world" assumption with a "locally-rigid" tiling approach, the model achieves state-of-the-art unsupervised depth and odometry results on the KITTI dataset and handles complex, non-rigid indoor scenes.

Introduction

Most unsupervised deep learning methods for depth and ego-motion estimation rely on view synthesis. The core principle is that if a model can predict the depth of a scene and the movement of the camera, it can mathematically "warp" one video frame to reconstruct the next. The difference between this synthetic frame and the actual next frame serves as the training signal.

However, these methods typically rely on a "rigid-world assumption"—the belief that the only thing moving in the scene is the camera. In reality, environments are filled with moving cars, pedestrians, and hands. When objects move independently, standard view synthesis equations break down, leading to distorted depth maps and inaccurate camera tracking. Previous solutions often required pre-trained semantic segmentation networks, which rely on human labels and defeat the purpose of a fully unsupervised system.

Methods

The authors propose a "locally-rigid" scene model based on the insight that while a whole scene might be non-rigid, small enough regions (like a patch on a moving car) are approximately rigid.

The Moving SLAM pipeline, showing depth estimation, local 6D motion prediction, and object segmentation.

Tiling Operator

Instead of predicting one global pose for the entire image, the network predicts a dense pose map. A "tiling" operator extracts small patches, each treated as a rigid entity with its own 6D motion (translation and rotation).

Unsupervised Segmentation

The system learns a mask ($M$) to partition the image into two categories:

  • Background: Uses a global rigid model, averaging the predicted poses of all background pixels.
  • Objects: Uses the local, non-rigid tiled model.

To prevent the model from "cheating" by marking the whole image as non-rigid (which has more degrees of freedom), the authors introduce a Constant Area Constraint. This soft constraint forces the network to assume that roughly the top 10% of pixels with the highest motion belong to moving objects.

Results and Evaluation

The model was evaluated on two distinct datasets: KITTI (autonomous driving) and EPIC-Kitchens (egocentric indoor videos).

Qualitative comparison of depth estimation on the KITTI dataset against existing unsupervised methods.

Key Findings

  • State-of-the-Art Odometry: On the KITTI odometry benchmark, the method achieved an Absolute Trajectory Error (ATE) of 0.011/0.010, outperforming both traditional SLAM pipelines (like ORB-SLAM) and previous unsupervised deep learning models.
  • Superior Depth Estimation: On KITTI, the model reached 88.9% accuracy (at $\delta < 1.25$), setting a new high bar for unsupervised monocular depth.
  • Generalization to Indoors: On EPIC-Kitchens, the model successfully recovered detailed depth maps and segmented moving hands and tools automatically, despite the lack of ground truth data or camera intrinsics.

Conclusion

Moving SLAM demonstrates that rigidity is a matter of scale. By shifting focus from the whole frame to sliding windows, the system models complex 6D object motions without complex new architectures. While limitations remain—such as the heuristic nature of the 10% foreground ratio and inherent monocular scale ambiguity—this "locally-rigid" approach paves the way for truly holistic SLAM algorithms that understand both the environment and the dynamic actors within it.

Summary updated

BibTeX
@inproceedings{xu2021moving,
  author = {Dan Xu and Andrea Vedaldi and João F. Henriques},
  title = {{Moving SLAM: Fully Unsupervised Deep Learning in Non-Rigid Scenes}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2021},
  doi = {10.1109/iros51168.2021.9636075},
}
Figure