Indoor Semantic Segmentation using depth information

Camille Couprie, Clement Farabet, Laurent Najman, Yann LeCun

Summary

This paper demonstrates that multiscale convolutional neural networks (ConvNets) can effectively learn to segment indoor scenes by treating depth information as an additional image channel. By moving away from hand-crafted features and leveraging RGB-D data, the authors achieved state-of-the-art accuracy on the NYU-v2 dataset and proved the approach is fast enough for near real-time applications.

Introduction and Motivation

Semantic segmentation—the task of assigning a category label (like "chair," "floor," or "wall") to every pixel in an image—is notoriously difficult in indoor environments. Unlike outdoor scenes, indoor spaces are often cluttered, unevenly lit, and filled with occluded objects.

While the release of the Microsoft Kinect made depth data (RGB-D) widely available, early attempts to use this data relied heavily on "hand-crafted" features—manually designed mathematical descriptors like SIFT. These methods were often computationally expensive and struggled to generalize across the messy reality of everyday rooms. The authors set out to see if a deep learning approach could "learn" better features directly from the raw data.

Architecture and Approach

The authors adapted a multiscale ConvNet architecture previously successful in outdoor scene parsing to accommodate the nuances of indoor RGB-D data.

Multiscale ConvNet Architecture

Key components of their approach include:

  • RGB-D Integration: The depth map is treated as a fourth channel, alongside Red, Green, and Blue. It is pre-processed so local neighborhoods have zero mean and unit standard deviation.
  • Multiscale Feature Extraction: The input is transformed into a Laplacian pyramid (multiple versions of the image at different resolutions). Each scale is fed into a 3-stage ConvNet to produce feature maps.
  • Contextual Concatenation: Coarser-scale maps are upsampled and concatenated with fine-scale maps. This allows the network to "see" both the fine details of an object and the broader context of the room simultaneously.
  • Superpixel Smoothing: To ensure the labels follow the natural boundaries of objects, the pixel-wise predictions from the ConvNet are aggregated within "superpixels" (clusters of similar pixels).
  • Temporal Consistency: For video processing, they employ temporally consistent superpixels to reduce "flickering" labels between frames.

Evaluation Datasets

The primary benchmark used is the NYU depth dataset version 2 (NYU-v2).

  • It contains 407,024 RGB-D frames from hundreds of video sequences.
  • 1,449 frames are densely labeled.
  • The authors evaluated their model using two different labeling schemes: a complex 14-class cluster and a broader 4-class semantic grouping (Ground, Furniture, Props, and Structure) to compare against previous state-of-the-art results.

Key Insights

  • Depth as a "Color": One of the most powerful ideas here is the simplicity of treating depth as just another channel. The ConvNet doesn't need a special geometric model of the world; it learns the spatial patterns of depth just as it learns the patterns of color.
  • The "Floor and Ceiling" Advantage: The inclusion of depth provides a massive boost (15% or more) for classes like "floor" and "ceiling." This is because these classes have very consistent depth gradients relative to the camera (which is usually held at chest height), making them easy for the network to identify regardless of their color or texture.
  • Feature Learning vs. Engineering: By learning features directly, the model outperformed previous methods that used complex, manually-engineered descriptors like 3D bounding box dimensions or surface normals.

Results and Performance

The integration of depth data yielded significant improvements in both quantitative metrics and qualitative visual outputs.

Qualitative Segmentation Results
  • Accuracy: The model achieved 64.5% pixel-wise accuracy on the NYU-v2 4-class task, surpassing the previous state-of-the-art (58.6%).
  • Class-wise Gains: On the 14-class task, adding depth information improved "ceiling" identification from 33.2% to 62.6% and "floor" from 68.0% to 87.3%.
  • Speed: The system is highly efficient, processing a 320x240 frame in approximately 0.7 seconds on a standard laptop. The authors note that with specialized hardware like an FPGA, this could easily reach real-time speeds.

Limitations and Future Directions

  • High-Variability Objects: The authors noted that for objects with high depth variability (like books or TVs), depth information can actually be less helpful than RGB alone. A future challenge is determining how to dynamically weight the importance of depth vs. color based on the object type.
  • Data Scarcity: Some errors occurred in video sequences when objects (like walls) appeared in the extreme foreground, a scenario underrepresented in the training data.
  • Refinement: The authors suggest that incorporating unsupervised feature learning or more advanced post-processing, such as Markov Random Fields (MRFs), could further sharpen the segmentation boundaries.

Summary updated

Abstract

This work addresses multi-class segmentation of indoor scenes with RGB-D inputs. While this area of research has gained much attention recently, most works still rely on hand-crafted features. In contrast, we apply a multiscale convolutional network to learn features directly from the images and the depth information. We obtain state-of-the-art on the NYU-v2 depth dataset with an accuracy of 64.5%. We illustrate the labeling of indoor scenes in videos sequences that could be processed in real-time using appropriate hardware such as an FPGA.
BibTeX
@inproceedings{couprie2013indoor,
  author = {Camille Couprie and Clement Farabet and Laurent Najman and Yann LeCun},
  title = {{Indoor Semantic Segmentation using depth information}},
  booktitle = {International Conference on Learning Representations},
  year = {2013},
  url = {https://openreview.net/forum?id=ttnAE7vaATtaK},
}

Loading related content...

Figure