Semantic Texture for Robust Dense Tracking.

Jan Czarnowski, Stefan Leutenegger, Andrew J. DavisonarXiv 2017 · OpenReview

TL;DR

This paper proposes replacing traditional image pyramids used in dense visual SLAM with feature maps from pre-trained Convolutional Neural Networks (CNNs). This "semantic texture" approach significantly improves tracking robustness against challenging lighting variations, demonstrating a promising new direction for real-time visual SLAM systems.

What Problem Does This Paper Address?

Dense visual Simultaneous Localization and Mapping (SLAM) systems aim to reconstruct a detailed 3D representation of a scene by aligning live camera images with a continuously updated model. A core challenge in this process is the "tracking" component, which relies on aligning current images with the existing scene model. Traditional methods often use raw RGB pixel values, which are highly sensitive to changes in lighting, shadows, and other environmental factors. This sensitivity can cause alignment to fail, leading to inaccurate reconstructions and tracking errors. While some research focuses on modeling lighting physics, this is computationally expensive. This paper addresses the need for a more robust and efficient way to represent scene appearance for tracking, specifically by finding a representation that is invariant to nuisance factors like lighting.

Methods

The authors propose replacing the traditional image pyramid (created by downsampling and blurring images) with a "semantic pyramid" derived from the feature maps of a pre-trained CNN (specifically, a VGG-16 network trained for image classification).

Here's how it works:

  • CNN Feature Pyramid: Instead of using blurred versions of the input image at different resolutions, the paper uses the outputs of successive convolutional layers of a CNN. These layers naturally form a hierarchy, with earlier layers capturing low-level features (edges, textures) and later layers capturing more complex, semantic information. This hierarchy acts as a "semantic texture" pyramid.
  • Lucas-Kanade Alignment: The core image alignment algorithm used is the standard Lucas-Kanade (LK) algorithm. The novelty lies in what is being aligned. Instead of aligning raw pixel values, the LK algorithm is applied to the multi-channel feature maps from the CNN.
  • Coarse-to-Fine Strategy: The alignment proceeds in a coarse-to-fine manner, starting with the feature maps from the earlier, lower-resolution CNN layers and progressively refining the alignment using features from deeper, higher-resolution layers.
  • Feature Selection: The paper also explores reducing computational cost by selecting a subset of CNN features. They propose criteria based on "texturedness" (using the smallest eigenvalue of the structure tensor) and "stability" (measuring how much feature activations change over time).

Datasets

The paper uses time-lapse sequences captured over many hours from static cameras for evaluation. Two specific sequences are mentioned:

  • "Window" sequence: Features a highly specular scene with major and irregular lighting changes.
  • "Home" sequence: Features a busier street with less specular surfaces and more global illumination changes.

These sequences are subsampled into 2-minute clips, and specific image pairs are selected to evaluate alignment performance under varying lighting conditions. They also use a sequence with a blurred version of an image to test robustness to blur.

Key Insights and Ideas

  • CNN Features as Robust Representations: The core insight is that CNN feature maps, even from networks trained for classification, inherently capture information that is more invariant to lighting changes than raw RGB values. This makes them ideal for robust visual tracking.
  • Bridging the Gap: The approach bridges the gap between traditional, handcrafted feature descriptors (like SIFT) and end-to-end learned methods. It leverages the robustness of learned features while retaining the accuracy of iterative optimization methods like Lucas-Kanade.
  • Semantic Texture for Alignment: The concept of "semantic texture" is powerful. It suggests that the hierarchical features learned by CNNs provide a richer, more semantically meaningful representation for aligning images than simple pixel intensities.
  • Feature Selection for Efficiency: The finding that a small, carefully selected subset of CNN features can achieve performance comparable to using all features is crucial for real-time applications. This suggests that future SLAM systems can be made significantly more efficient without sacrificing accuracy.

Key Results

  • Improved Robustness to Lighting: The proposed CNN pyramid approach consistently outperforms raw RGB and dense SIFT methods in terms of the size of the convergence basin, especially under challenging lighting conditions. This means the alignment is more likely to succeed even with significant illumination changes.
  • Comparable Performance with Fewer Features: By selecting features based on texturedness and stability, the authors achieve performance comparable to using the full set of CNN features, but with a significantly reduced number of features. This is demonstrated in Figure 9, where their selection method achieves high convergence basin sizes with much lower percentages of features compared to random selection.
  • Real-time Performance: The system is capable of real-time performance (15-20 frames per second) for tracking, with feature extraction taking around 1ms per frame and alignment taking 40-60ms.

Limitations and Future Work

  • Computational Cost of Full Feature Set: While the method is robust, using all CNN features can be computationally expensive. The paper addresses this by exploring feature selection, but further optimization might be needed.
  • Generalization of CNNs: The paper notes that while any classification CNN could be used, they only tested VGG-16. The performance might vary with different architectures or training datasets.
  • Future Work:
    • Investigating "m-estimators" on CNN features to better handle outliers.
  • Exploring whether semantic regions that have moved or changed can be masked out from tracking.
  • Developing dense 3D models painted with learned feature textures rather than raw pixel values.
  • Further research into learning ideal feature sets for specific SLAM tasks.

Summary updated

Abstract

We argue that robust dense SLAM systems can make valuable use of the layers of features coming from a standard CNN as a pyramid of `semantic texture' which is suitable for dense alignment while being much more robust to nuisance factors such as lighting than raw RGB values. We use a straightforward Lucas-Kanade formulation of image alignment, with a schedule of iterations over the coarse-to-fine levels of a pyramid, and simply replace the usual image pyramid by the hierarchy of convolutional feature maps from a pre-trained CNN. The resulting dense alignment performance is much more robust to lighting and other variations, as we show by camera rotation tracking experiments on time-lapse sequences captured over many hours. Looking towards the future of scene representation for real-time visual SLAM, we further demonstrate that a selection using simple criteria of a small number of the total set of features output by a CNN gives just as accurate but much more efficient tracking performance.
BibTeX
@misc{czarnowski2017semantic,
  author = {Jan Czarnowski and Stefan Leutenegger and Andrew J. Davison},
  title = {{Semantic Texture for Robust Dense Tracking.}},
  year = {2017},
  eprint = {1708.08844},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1708.08844},
}
Figure