Indoor Scene Recognition in 3D

Shengyu Huang, Mikhail Usvyatsov, Konrad Schindler

TL;DR

This paper demonstrates that 3D point cloud data significantly outperforms 2D methods for indoor scene recognition. They achieve state-of-the-art results by employing multi-task learning, where semantic segmentation is used as an auxiliary task to improve scene classification, even with surprisingly sparse 3D data.

What Problem Does This Paper Address?

For robots and autonomous systems operating indoors, understanding the environment (e.g., knowing if it's a kitchen, bedroom, or hallway) is crucial for effective navigation and task execution. While previous work focused on 2D images or 2.5D range data, this paper tackles the challenge of indoor scene recognition directly from 3D point clouds or voxel representations. The authors highlight that 3D data is a more natural fit for indoor environments and that existing machine learning techniques for 3D are maturing, yet 3D scene recognition has been underexplored, partly due to a lack of suitable public datasets.

Methods

The core of the paper's methodology revolves around leveraging 3D point cloud data for scene recognition. They propose a multi-task learning framework where scene classification and semantic segmentation are trained jointly.

  • Shared Encoder: A single encoder network processes the 3D input (either subsampled point clouds or sparse voxel grids) to generate a latent representation.
  • Dual Decoders: This latent representation is then fed into two separate decoder branches: one for scene classification and another for semantic segmentation.
  • Sparse 3D Architectures: To handle the computational demands of 3D data, they utilize efficient sparse convolutional neural networks, specifically a variant of ResNet14 with a U-Net-style decoder for semantic segmentation.
  • Optimization Strategy: A novel optimization approach is introduced: first, the network is trained for semantic segmentation (which has richer supervision), then the encoder weights are frozen to train the scene classification head, and finally, the entire network is fine-tuned end-to-end with a low learning rate. This strategy helps to effectively leverage the strong supervision from semantic segmentation to improve scene classification.

Datasets

The primary dataset used in this paper is ScanNet.

  • ScanNet is an RGB-D video dataset containing over 1500 scans of indoor environments.
  • The authors use 1013 scans for training and 500 for validation.
  • The benchmark dataset includes 21 different scene types, though only 13 are evaluated in the paper's main experiments.

Key Insights and Ideas

  • 3D Dominance: 3D point cloud data provides a significant performance boost for indoor scene recognition compared to 2D methods, even outperforming 2D birds-eye views.
  • Multi-task Learning Synergy: Jointly training for semantic segmentation and scene classification is highly effective. The rich, per-point labels for semantic segmentation provide valuable "side information" that guides the learning of more discriminative features for scene classification.
  • Sparsity is Sufficient: Surprisingly, very sparse 3D point clouds (as few as 128 points) are sufficient to achieve good scene classification accuracy, suggesting that a quick "glance" at the environment is often enough.
  • Beyond "Marker" Objects: Scene recognition is not solely about identifying individual distinctive objects (like a bathtub in a bathroom). It relies on a combination of coarse 3D geometry, color, and the distribution of object categories. The overall scene shape plays a crucial role.
  • Geometry vs. Semantics Trade-off: While geometry alone can achieve high accuracy with sparse data, achieving the absolute best performance requires dense point clouds and semantic segmentation, highlighting the complementary nature of these cues.

Key Results

  • The proposed multi-task learning approach achieves a state-of-the-art 90.3% accuracy on the ScanNet validation set.
  • This significantly outperforms previous 2D methods on the ScanNet benchmark, achieving 70.0% recall and 64.6% mean IoU, compared to prior submissions of around 49.8% recall and 35.5% mean IoU.
  • Even with only 128 points, Pointnet++ and ResNet14 achieve over 75% accuracy, with performance saturating around 85% with 512-1024 points.
  • The multi-task approach, leveraging both geometry and semantic information, boosts performance significantly compared to geometry-only or semantics-only baselines.

Limitations and Future Work

  • Completeness Matters: While sparse data is sufficient for good accuracy, the performance degrades significantly with incomplete scenes (e.g., <58% accuracy at 50% crop ratio). The authors suggest that capturing a coarse but complete 3D view is preferable to a detailed but partial one.
  • Auxiliary Tasks for Sparsity: The paper notes that while multi-task learning with semantic segmentation is beneficial, it requires high point density. An open question is whether other auxiliary tasks can be found that are effective at low point densities.
  • Alternative Representations: The authors suggest exploring alternative 3D representations that efficiently capture global scene structure, such as sparse depth panoramas, as a potential avenue for future research.
  • Data Augmentation: While data augmentation was used, the authors suggest that more sophisticated schemes might improve robustness against incomplete scenes.

Summary updated

Abstract

Recognising in what type of environment one is located is an important perception task. For instance, for a robot operating in indoors it is helpful to be aware whether it is in a kitchen, a hallway or a bedroom. Existing approaches attempt to classify the scene based on 2D images or 2.5D range images. Here, we study scene recognition from 3D point cloud (or voxel) data, and show that it greatly outperforms methods based on 2D birds-eye views. Moreover, we advocate multi-task learning as a way of improving scene recognition, building on the fact that the scene type is highly correlated with the objects in the scene, and therefore with its semantic segmentation into different object classes. In a series of ablation studies, we show that successful scene recognition is not just the recognition of individual objects unique to some scene type (such as a bathtub), but depends on several different cues, including coarse 3D geometry, colour, and the (implicit) distribution of object categories. Moreover, we demonstrate that surprisingly sparse 3D data is sufficient to classify indoor scenes with good accuracy.
BibTeX
@inproceedings{huang2020indoor,
  author = {Shengyu Huang and Mikhail Usvyatsov and Konrad Schindler},
  title = {{Indoor Scene Recognition in 3D}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2020},
  doi = {10.1109/iros45743.2020.9341580},
}

Loading related content...

Figure