Moving SLAM: Fully Unsupervised Deep Learning in Non-Rigid Scenes

Dan Xu, Andrea Vedaldi, João F. Henriques

Abstract

This paper introduces "Moving SLAM," a framework that learns to decompose videos into 3D geometry, camera motion, and moving objects without any human supervision. By replacing the standard "rigid-world" assumption with a "locally-rigid" tiling approach, the model achieves state-of-the-art unsupervised depth and odometry results on the KITTI dataset and handles complex, non-rigid indoor scenes.

Introduction

Most unsupervised deep learning methods for depth and ego-motion estimation rely on view synthesis. The core principle is that if a model can predict the depth of a scene and the movement of the camera, it can mathematically "warp" one video frame to reconstruct the next. The difference between this synthetic frame and the actual next frame serves as the training signal.

However, these methods typically rely on a "rigid-world assumption"—the belief that the only thing moving in the scene is the camera. In reality, environments are filled with moving cars, pedestrians, and hands. When objects move independently, standard view synthesis equations break down, leading to distorted depth maps and inaccurate camera tracking. Previous solutions often required pre-trained semantic segmentation networks, which rely on human labels and defeat the purpose of a fully unsupervised system.

Methods

The authors propose a "locally-rigid" scene model based on the insight that while a whole scene might be non-rigid, small enough regions (like a patch on a moving car) are approximately rigid.

The Moving SLAM pipeline, showing depth estimation, local 6D motion prediction, and object segmentation.

Tiling Operator

Instead of predicting one global pose for the entire image, the network predicts a dense pose map. A "tiling" operator extracts small patches, each treated as a rigid entity with its own 6D motion (translation and rotation).

Unsupervised Segmentation

The system learns a mask ($M$) to partition the image into two categories:

  • Background: Uses a global rigid model, averaging the predicted poses of all background pixels.
  • Objects: Uses the local, non-rigid tiled model.

To prevent the model from "cheating" by marking the whole image as non-rigid (which has more degrees of freedom), the authors introduce a Constant Area Constraint. This soft constraint forces the network to assume that roughly the top 10% of pixels with the highest motion belong to moving objects.

Results and Evaluation

The model was evaluated on two distinct datasets: KITTI (autonomous driving) and EPIC-Kitchens (egocentric indoor videos).

Qualitative comparison of depth estimation on the KITTI dataset against existing unsupervised methods.

Key Findings

  • State-of-the-Art Odometry: On the KITTI odometry benchmark, the method achieved an Absolute Trajectory Error (ATE) of 0.011/0.010, outperforming both traditional SLAM pipelines (like ORB-SLAM) and previous unsupervised deep learning models.
  • Superior Depth Estimation: On KITTI, the model reached 88.9% accuracy (at $\delta < 1.25$), setting a new high bar for unsupervised monocular depth.
  • Generalization to Indoors: On EPIC-Kitchens, the model successfully recovered detailed depth maps and segmented moving hands and tools automatically, despite the lack of ground truth data or camera intrinsics.

Conclusion

Moving SLAM demonstrates that rigidity is a matter of scale. By shifting focus from the whole frame to sliding windows, the system models complex 6D object motions without complex new architectures. While limitations remain—such as the heuristic nature of the 10% foreground ratio and inherent monocular scale ambiguity—this "locally-rigid" approach paves the way for truly holistic SLAM algorithms that understand both the environment and the dynamic actors within it.

Summary updated

Abstract

We propose a method to train deep networks to decompose videos into 3D geometry (camera and depth), moving objects, and their motions, with no supervision. We build on the idea of view synthesis, which uses classical camera geometry to re-render a source image from a different point-of-view, specified by a predicted relative pose and depth map. By minimizing the error between the synthetic image and the corresponding real image in a video, the deep network that predicts pose and depth can be trained completely unsupervised. However, the view synthesis equations rely on a strong assumption: that objects do not move. This rigid-world assumption limits the predictive power, and rules out learning about objects automatically. We propose a simple solution: minimize the error on small regions of the image instead. While the scene as a whole may be non-rigid, it is always possible to find small regions that are approximately rigid, such as inside a moving object. Our network can then predict different poses for each region, in a sliding window from a learned dense pose map. This represents a significantly richer model, including 6D object motions, with little additional complexity. We achieve very competitive performance on unsupervised odometry and depth prediction on KITTI. We also demonstrate new capabilities on EPIC-Kitchens, a challenging dataset of indoor videos, where there is no ground truth information for depth, odometry, object segmentation or motion. Yet all are recovered automatically by our method.
BibTeX
@inproceedings{xu2021moving,
  author = {Dan Xu and Andrea Vedaldi and João F. Henriques},
  title = {{Moving SLAM: Fully Unsupervised Deep Learning in Non-Rigid Scenes}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2021},
  doi = {10.1109/iros51168.2021.9636075},
}
Figure