Abstract
Standard deep learning models for semantic segmentation often treat video frames as isolated events, leading to temporal inconsistencies and "flickering" predictions. This research from the Technical University of Munich presents a framework that enforces multi-view consistency during training. By utilizing camera trajectories from RGB-D SLAM, the system warps features from neighboring views into a common reference frame. This geometric constraint allows the network to learn robust, view-invariant features, achieving state-of-the-art results on the NYUDv2 dataset for both single-view and multi-view fusion tasks.
Introduction
In robotics, understanding a 3D environment requires more than static image analysis. While deep learning has significantly advanced semantic segmentation, most models struggle with consistency across different viewpoints. A pixel labeled as a "table" in one frame might be misidentified as a "desk" in the next due to changes in lighting or perspective. Furthermore, the high cost of per-frame manual annotation limits the availability of large-scale training data for video sequences.
This paper addresses these challenges by leveraging the natural redundancy in video sequences through geometric constraints. By enforcing consistency across multiple views, the network learns to recognize objects regardless of the camera's pose.
Methodology
The proposed architecture, MVCNet (Multi-View Consistent Network), builds upon the FuseNet encoder-decoder structure, which integrates RGB and depth information.
Multi-Scale Deep Supervision
To improve detail refinement, the authors implement multi-scale deep supervision. Instead of calculating loss only at the final output, supervision is applied at every stage of the decoder. This encourages the network to learn hierarchical features that progressively restore spatial resolution.
Multi-View Consistency Training
The core innovation lies in the use of RGB-D SLAM to track camera trajectories. During training, the network processes a reference frame alongside several neighboring frames. Using known 3D geometry, features or predictions from these neighbors are warped into the reference frame's perspective.
The authors explored three strategies for enforcing consistency:
- Warp Augmentation: Using warped views as additional training data.
- Bayesian Fusion: Combining final probability maps from multiple views.
- Multi-View Max-Pooling: Selecting the most salient features across views at the feature-map level. The authors found that feature-level fusion via max-pooling yielded the most robust performance.
Experimental Results
The framework was evaluated on the NYUDv2 benchmark for both 13-class and 40-class segmentation tasks. Camera poses were generated using DVO SLAM on raw sequences.
- Single-View Performance: Training with multi-view constraints significantly improved single-image accuracy. On the 40-class task, MVCNet achieved 69.1% pixelwise accuracy, surpassing previous state-of-the-art models like FuseNet and Context-CRF.
- Multi-View Fusion: Accuracy further increased when fusing predictions from 50 frames at test time. For the 13-class task, the model reached 79.13% pixelwise accuracy and a 59.07% IoU.
- Qualitative Analysis: The resulting maps exhibited higher homogeneity and better alignment with physical object boundaries compared to baseline models.
Conclusion
By integrating geometry as a supervisor, this work demonstrates that multi-view consistency is a powerful tool for training robust semantic segmentation networks. The approach reduces the need for exhaustive manual labeling by exploiting the inherent structure of video data. Future work will focus on addressing limitations related to mirror reflections and achieving real-time integration into full semantic SLAM systems.
Summary updated