Toward Geometric Deep SLAM

Daniel DeTone, Tomasz Malisiewicz, Andrew RabinovicharXiv 2017 · arXiv

TL;DR

Researchers from Magic Leap propose a real-time, CPU-friendly point tracking system using two specialized neural networks: MagicPoint for corner detection and MagicWarp for homography estimation. By training exclusively on simple synthetic shapes with heavy noise injection, the system achieves state-of-the-art robustness on real-world data without the need for expensive, manually labeled SLAM datasets.

What Problem Does This Paper Address?

Deep learning has revolutionized computer vision, but Simultaneous Localization and Mapping (SLAM) remains a difficult frontier. The primary bottleneck is data: unlike object detection, which benefits from massive datasets like ImageNet, SLAM requires precise 6DoF ground-truth poses across diverse environments. Obtaining this data is expensive, difficult to scale, and often device-specific.

Furthermore, while photorealistic rendering offers a potential solution, models trained on high-end graphics often fail to generalize to the messy, noisy reality of low-cost sensors. Traditional point detectors (like FAST or Harris corners) are fast but brittle under sensor noise and lighting changes. The authors ask: Can we build a "Deep SLAM" system that is lean enough for embedded hardware and robust enough to learn from simple synthetic data?

Methods

The authors decompose the tracking problem into two distinct stages, each handled by a dedicated Convolutional Neural Network (CNN):

  1. MagicPoint (Detection): This network takes a single grayscale image and outputs a "point-ness" probability map. It uses a VGG-style encoder to downsample the image. To keep the model "lean," the authors avoid expensive upconvolutional layers. Instead, they use an explicit decoder (sub-pixel convolution) where the final tensor represents an 8x8 grid of pixels plus a "dustbin" channel for "no point detected."
  2. MagicWarp (Transformation): Instead of using traditional local descriptors (like SIFT or ORB), MagicWarp operates solely on the point locations produced by MagicPoint. It takes a pair of point images, concatenates them, and regresses a 3x3 homography matrix ($H$). This approach forces the network to focus on geometric consistency rather than photometric texture.
Figure 1: Deep Point-Based Tracking Overview. Pairs of images are processed by a convolutional neural network called MagicPoint which is trained to detect salient corners in the image. The resulting point images are then processed together by MagicWarp (another convolutional neural network) to compute a homography H which relates the points in the input images.

Datasets

A key contribution of the paper is the Synthetic Shapes Dataset. Rather than using complex real-world scenes, the authors trained their networks on a simple renderer that generates triangles, quadrilaterals, lines, cubes, and checkerboards.

  • Training: Millions of synthetic examples were generated on-the-fly, augmented with homographic warping and extreme noise (Gaussian, speckle, shadows, and motion blur).
  • Evaluation: The system was tested on the 30 Static Corners Dataset, a new real-world benchmark consisting of 30 video sequences of simple geometric scenes under varying lighting conditions, recorded with a standard webcam.
Figure 3: Synthetic Shapes Dataset. The Synthetic Shapes dataset consists of rendered triangles, quadrilaterals, lines, cubes, checkerboards, and stars each with ground truth corner locations. It also includes some negative images with no ground truth corners, such as ellipses and random noise images.

Key Insights and Ideas

  • Synthetic-to-Real Transfer: The most surprising finding is that a network trained only on simple 2D shapes can effectively detect corners in the real world. The "low-level" nature of corners means that complex textures aren't necessary for training; geometric primitives are sufficient.
  • Noise as a Teacher: By injecting massive amounts of noise during synthetic training, the authors created a detector that significantly outperforms classical hand-engineered filters (FAST, Harris) when image quality degrades.
  • Geometry over Photometry: MagicWarp's success suggests that point distributions alone contain enough information to estimate motion, bypassing the need for complex, high-dimensional point descriptors that are often sensitive to lighting changes.
  • Efficiency by Design: By using a "depth-to-space" reshape instead of standard decoders, MagicPoint remains incredibly fast, running at over 30 FPS on a single CPU—a necessity for mobile AR and robotics.
The MagicPoint architecture featuring the efficient explicit decoder.

Key Results

  • Robustness: In the presence of noise, MagicPoint maintained a Mean Average Precision (mAP) of ~0.97, while classical detectors like FAST dropped to near zero (0.06).
  • Localization: MagicPoint achieved a Mean Localization Error (MLE) of roughly 1 pixel, outperforming Harris and Shi-Tomasi baselines on both synthetic and real data.
  • Tracking: MagicWarp significantly outperformed Nearest Neighbor matching. For medium-density point sets with 20% noise, MagicWarp could handle rotations up to 18° and translations up to 18 pixels while maintaining 90% match repeatability, whereas Nearest Neighbor failed at just 5° and 5 pixels.
  • Speed: The small version of MagicPoint (81KB) takes only 5.3ms for a 160x120 image on a CPU.

Limitations and Future Work

The current system is primarily designed for homography estimation, which assumes planar scenes or distant cameras. While this is useful for many AR tasks, it does not yet represent a full 3D SLAM system capable of handling complex, non-planar environments (which would require estimating a Fundamental or Essential matrix).

The authors suggest that future work will involve moving toward a full end-to-end Deep SLAM system that incorporates depth estimation and handles more complex 3D geometries, potentially still leveraging the power of synthetic-to-real training.

Summary updated

Abstract

We present a point tracking system powered by two deep convolutional neural networks. The first network, MagicPoint, operates on single images and extracts salient 2D points. The extracted points are "SLAM-ready" because they are by design isolated and well-distributed throughout the image. We compare this network against classical point detectors and discover a significant performance gap in the presence of image noise. As transformation estimation is more simple when the detected points are geometrically stable, we designed a second network, MagicWarp, which operates on pairs of point images (outputs of MagicPoint), and estimates the homography that relates the inputs. This transformation engine differs from traditional approaches because it does not use local point descriptors, only point locations. Both networks are trained with simple synthetic data, alleviating the requirement of expensive external camera ground truthing and advanced graphics rendering pipelines. The system is fast and lean, easily running 30+ FPS on a single CPU.
BibTeX
@misc{detone2017toward,
  author = {Daniel DeTone and Tomasz Malisiewicz and Andrew Rabinovich},
  title = {{Toward Geometric Deep SLAM}},
  year = {2017},
  eprint = {1707.07410},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1707.07410},
}

Loading related content...

Figure