Multi-view deep learning for consistent semantic mapping with RGB-D cameras

Lingni Ma, Jörg Stückler, Christian Kerl, Daniel Cremers

Abstract

Standard deep learning models for semantic segmentation often treat video frames as isolated events, leading to temporal inconsistencies and "flickering" predictions. This research from the Technical University of Munich presents a framework that enforces multi-view consistency during training. By utilizing camera trajectories from RGB-D SLAM, the system warps features from neighboring views into a common reference frame. This geometric constraint allows the network to learn robust, view-invariant features, achieving state-of-the-art results on the NYUDv2 dataset for both single-view and multi-view fusion tasks.

Introduction

In robotics, understanding a 3D environment requires more than static image analysis. While deep learning has significantly advanced semantic segmentation, most models struggle with consistency across different viewpoints. A pixel labeled as a "table" in one frame might be misidentified as a "desk" in the next due to changes in lighting or perspective. Furthermore, the high cost of per-frame manual annotation limits the availability of large-scale training data for video sequences.

The proposed framework enforces consistency by warping neighboring views into a reference frame for supervision.

This paper addresses these challenges by leveraging the natural redundancy in video sequences through geometric constraints. By enforcing consistency across multiple views, the network learns to recognize objects regardless of the camera's pose.

Methodology

The proposed architecture, MVCNet (Multi-View Consistent Network), builds upon the FuseNet encoder-decoder structure, which integrates RGB and depth information.

Multi-Scale Deep Supervision

To improve detail refinement, the authors implement multi-scale deep supervision. Instead of calculating loss only at the final output, supervision is applied at every stage of the decoder. This encourages the network to learn hierarchical features that progressively restore spatial resolution.

Multi-View Consistency Training

The core innovation lies in the use of RGB-D SLAM to track camera trajectories. During training, the network processes a reference frame alongside several neighboring frames. Using known 3D geometry, features or predictions from these neighbors are warped into the reference frame's perspective.

Detailed MVCNet architecture showing RGB-D fusion, multi-scale deconvolution, and the multi-view consistency supervision module.

The authors explored three strategies for enforcing consistency:

  1. Warp Augmentation: Using warped views as additional training data.
  2. Bayesian Fusion: Combining final probability maps from multiple views.
  3. Multi-View Max-Pooling: Selecting the most salient features across views at the feature-map level. The authors found that feature-level fusion via max-pooling yielded the most robust performance.

Experimental Results

The framework was evaluated on the NYUDv2 benchmark for both 13-class and 40-class segmentation tasks. Camera poses were generated using DVO SLAM on raw sequences.

  • Single-View Performance: Training with multi-view constraints significantly improved single-image accuracy. On the 40-class task, MVCNet achieved 69.1% pixelwise accuracy, surpassing previous state-of-the-art models like FuseNet and Context-CRF.
  • Multi-View Fusion: Accuracy further increased when fusing predictions from 50 frames at test time. For the 13-class task, the model reached 79.13% pixelwise accuracy and a 59.07% IoU.
  • Qualitative Analysis: The resulting maps exhibited higher homogeneity and better alignment with physical object boundaries compared to baseline models.

Conclusion

By integrating geometry as a supervisor, this work demonstrates that multi-view consistency is a powerful tool for training robust semantic segmentation networks. The approach reduces the need for exhaustive manual labeling by exploiting the inherent structure of video data. Future work will focus on addressing limitations related to mirror reflections and achieving real-time integration into full semantic SLAM systems.

Summary updated

BibTeX
@inproceedings{ma2017multi,
  author = {Lingni Ma and Jörg Stückler and Christian Kerl and Daniel Cremers},
  title = {{Multi-view deep learning for consistent semantic mapping with RGB-D cameras}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2017},
  doi = {10.1109/iros.2017.8202213},
}

Loading related content...

Figure