Learning Effective RGB-D Representations for Scene Recognition

Xinhang Song, Shuqiang Jiang, Luis Herranz, Chengpeng Chen

Abstract

Deep convolutional networks (CNN) have achieved impressive results on RGB scene recognition, but RGB-D recognition remains underdeveloped due to data scarcity and the limited range of depth sensors. This paper addresses these issues by proposing a two-step training approach that learns depth-specific features from scratch using weakly-supervised patches. Furthermore, the authors introduce the ISIA RGB-D video dataset and a CNN-RNN architecture to capture more comprehensive depth information by accumulating temporal data as a camera moves through a scene. The approach achieves state-of-the-art performance on standard benchmarks and the new video dataset.

What Problem Does This Paper Address?

While RGB scene recognition benefits from massive datasets like Places, RGB-D recognition faces two primary hurdles:

  1. Data Scarcity: Depth datasets are significantly smaller than RGB ones. Standard practice involves fine-tuning RGB-pretrained models, but depth images possess distinct statistical properties (edges and shapes) compared to RGB (textures and colors). RGB-trained filters often fail to capture these unique depth patterns effectively.
  2. Sensor Range Limitations: Common depth sensors, such as the Microsoft Kinect, have a limited functional range (typically 0.5m to 5.5m). In large environments, distant objects appear as missing data (black holes) in depth maps. This can lead to classification ambiguity; for instance, a large furniture store might be misidentified as a bedroom if distant shelves are invisible.
Comparison of RGB and Depth images across different scenes, highlighting the range limitations of depth sensors (black regions).

Methods

The authors propose a multi-pronged strategy to overcome these limitations:

Weakly-Supervised Patch Pretraining

To address data scarcity without relying on RGB weights, the authors train a depth-specific CNN (D-CNN) from scratch. They augment the training set by breaking images into patches and assigning each patch the label of its parent image. This "weak supervision" allows the network to learn fundamental mid-level depth features, such as walls, floors, and furniture edges, from a larger volume of samples.

D-CNN Architecture

The D-CNN uses a smaller, more efficient architecture tailored for depth data. This design is less prone to overfitting on small datasets compared to massive models like VGG or AlexNet.

CNN-RNN for Video Recognition

To mitigate range limitations, the authors transition from static images to video. As a camera traverses a scene, it eventually captures depth information for objects that were previously too distant. They employ a Recurrent Neural Network (LSTM) to aggregate these temporal "glimpses" into a robust spatio-temporal representation.

Multimodal Fusion

The final system employs a late-fusion approach, combining the outputs of an RGB branch (pretrained on Places) and the custom Depth branch through fully connected layers to leverage complementary features.

Datasets

The study evaluates its approach on three benchmarks:

  • NYUD2: A standard indoor dataset with 27 categories.
  • SUN RGB-D: A large-scale benchmark with over 10,000 images across 19 categories.
  • ISIA RGB-D (New): A video dataset introduced in this work, containing 278 videos across 58 categories, captured in diverse urban environments.

Key Results and Insights

The authors demonstrate that learning depth features from scratch is superior to fine-tuning RGB models. On the SUN RGB-D dataset, their D-CNN achieved 41.2% accuracy using only depth features, outperforming a standard fine-tuned Places-CNN (37.5%).

When combining modalities, the model reached 52.4% on SUN RGB-D and 67.5% on NYUD2, setting new state-of-the-art benchmarks. Furthermore, on the ISIA RGB-D dataset, the video-based approach (CNN+LSTM) provided a 6.3% mAP gain over single-frame recognition, confirming that temporal accumulation effectively acts as a range extender for depth sensors.

Conclusion

By addressing the fundamental limitations of depth data through patch-based pretraining and temporal video aggregation, this work provides a more effective framework for RGB-D scene recognition. The results highlight the importance of depth-specific feature learning and the potential of video data to overcome hardware constraints in spatial sensing.

Summary updated

Abstract

Deep convolutional networks (CNN) can achieve impressive results on RGB scene recognition thanks to large datasets such as Places. In contrast, RGB-D scene recognition is still underdeveloped in comparison, due to two limitations of RGB-D data we address in this paper. The first limitation is the lack of depth data for training deep learning models. Rather than fine tuning or transferring RGB-specific features, we address this limitation by proposing an architecture and a two-step training approach that directly learns effective depth-specific features using weak supervision via patches. The resulting RGB-D model also benefits from more complementary multimodal features. Another limitation is the short range of depth sensors (typically 0.5m to 5.5m), resulting in depth images not capturing distant objects in the scenes that RGB images can. We show that this limitation can be addressed by using RGB-D videos, where more comprehensive depth information is accumulated as the camera travels across the scene. Focusing on this scenario, we introduce the ISIA RGB-D video dataset to evaluate RGB-D scene recognition with videos. Our video recognition architecture combines convolutional and recurrent neural networks (RNNs) that are trained in three steps with increasingly complex data to learn effective features (i.e. patches, frames and sequences). Our approach obtains state-of-the-art performances on RGB-D image (NYUD2 and SUN RGB-D) and video (ISIA RGB-D) scene recognition.
BibTeX
@article{song2019learning,
  author = {Xinhang Song and Shuqiang Jiang and Luis Herranz and Chengpeng Chen},
  title = {{Learning Effective RGB-D Representations for Scene Recognition}},
  journal = {IEEE Trans. Image Process.},
  year = {2019},
  url = {https://openreview.net/forum?id=mFdMegMS4C},
}
Figure