Abstract
Deep convolutional networks (CNN) have achieved impressive results on RGB scene recognition, but RGB-D recognition remains underdeveloped due to data scarcity and the limited range of depth sensors. This paper addresses these issues by proposing a two-step training approach that learns depth-specific features from scratch using weakly-supervised patches. Furthermore, the authors introduce the ISIA RGB-D video dataset and a CNN-RNN architecture to capture more comprehensive depth information by accumulating temporal data as a camera moves through a scene. The approach achieves state-of-the-art performance on standard benchmarks and the new video dataset.
What Problem Does This Paper Address?
While RGB scene recognition benefits from massive datasets like Places, RGB-D recognition faces two primary hurdles:
- Data Scarcity: Depth datasets are significantly smaller than RGB ones. Standard practice involves fine-tuning RGB-pretrained models, but depth images possess distinct statistical properties (edges and shapes) compared to RGB (textures and colors). RGB-trained filters often fail to capture these unique depth patterns effectively.
- Sensor Range Limitations: Common depth sensors, such as the Microsoft Kinect, have a limited functional range (typically 0.5m to 5.5m). In large environments, distant objects appear as missing data (black holes) in depth maps. This can lead to classification ambiguity; for instance, a large furniture store might be misidentified as a bedroom if distant shelves are invisible.
Methods
The authors propose a multi-pronged strategy to overcome these limitations:
Weakly-Supervised Patch Pretraining
To address data scarcity without relying on RGB weights, the authors train a depth-specific CNN (D-CNN) from scratch. They augment the training set by breaking images into patches and assigning each patch the label of its parent image. This "weak supervision" allows the network to learn fundamental mid-level depth features, such as walls, floors, and furniture edges, from a larger volume of samples.
D-CNN Architecture
The D-CNN uses a smaller, more efficient architecture tailored for depth data. This design is less prone to overfitting on small datasets compared to massive models like VGG or AlexNet.
CNN-RNN for Video Recognition
To mitigate range limitations, the authors transition from static images to video. As a camera traverses a scene, it eventually captures depth information for objects that were previously too distant. They employ a Recurrent Neural Network (LSTM) to aggregate these temporal "glimpses" into a robust spatio-temporal representation.
Multimodal Fusion
The final system employs a late-fusion approach, combining the outputs of an RGB branch (pretrained on Places) and the custom Depth branch through fully connected layers to leverage complementary features.
Datasets
The study evaluates its approach on three benchmarks:
- NYUD2: A standard indoor dataset with 27 categories.
- SUN RGB-D: A large-scale benchmark with over 10,000 images across 19 categories.
- ISIA RGB-D (New): A video dataset introduced in this work, containing 278 videos across 58 categories, captured in diverse urban environments.
Key Results and Insights
The authors demonstrate that learning depth features from scratch is superior to fine-tuning RGB models. On the SUN RGB-D dataset, their D-CNN achieved 41.2% accuracy using only depth features, outperforming a standard fine-tuned Places-CNN (37.5%).
When combining modalities, the model reached 52.4% on SUN RGB-D and 67.5% on NYUD2, setting new state-of-the-art benchmarks. Furthermore, on the ISIA RGB-D dataset, the video-based approach (CNN+LSTM) provided a 6.3% mAP gain over single-frame recognition, confirming that temporal accumulation effectively acts as a range extender for depth sensors.
Conclusion
By addressing the fundamental limitations of depth data through patch-based pretraining and temporal video aggregation, this work provides a more effective framework for RGB-D scene recognition. The results highlight the importance of depth-specific feature learning and the potential of video data to overcome hardware constraints in spatial sensing.
Summary updated