RGB-D Scene Recognition via Spatial-Related Multi-Modal Feature Learning

Zhitong Xiong, Yuan Yuan, Qi WangIEEE Access 2019 · DOI

Introduction

Scene recognition is a fundamental challenge in computer vision that differs significantly from standard object recognition. While identifying a "dog" is often object-centric, identifying a "living room" requires understanding the global layout and the spatial arrangement of various objects within an environment. This complexity is further compounded when using RGB-D data, which combines color (RGB) and geometric (Depth) information.

Sample scenes illustrating the diversity and complexity of environments in scene recognition tasks, where identifying the overall context is as important as identifying individual objects.

The Challenge of Multi-Modal Scene Understanding

Current approaches to RGB-D scene recognition face two primary hurdles:

  1. Multi-modal Fusion: Effectively combining color data with geometric depth information to create a unified representation.
  2. Local Feature Extraction: Capturing discriminative local details (like a stove in a kitchen) without the computational overhead of dense sampling or the need for expensive, manually-labeled object-level annotations.

Standard Convolutional Neural Networks (CNNs) often struggle with the high spatial variability of scenes when they rely solely on global features. The authors sought a way to adaptively find the most important parts of a scene to boost classification accuracy.

Proposed Methodology

The authors introduce a two-branch CNN architecture designed to learn both global scene layouts and "key" local features simultaneously.

Key Feature Selection (KFS) Module

The core innovation is the KFS module, which employs a spatial attention mechanism based on non-local networks. It adaptively identifies high-response regions in feature maps—representing scene "themes"—and selects the top K local feature vectors. This allows for weakly supervised localization of important scene elements without requiring explicit object labels.

Novel Loss Functions

To optimize the multi-modal learning process, the framework introduces two specific loss functions:

  • Spatial-Attention Similarity Loss ($L_{sim}$): This loss forces the attention masks of the RGB and Depth branches to align. Since both sensors observe the same physical space, this alignment helps the depth branch ignore sensor noise by "following the lead" of the cleaner RGB signal.
  • Triplet Correlation Loss: This ensures that local features from images within the same category are highly correlated, while those from different categories remain distinct.

Experimental Results

The model was evaluated on two major indoor scene benchmarks: SUN RGB-D (10,355 images across 19 categories) and NYU Depth v2 (1,449 images across 10 categories).

The proposed method achieved state-of-the-art (SOTA) performance at the time of publication:

  • SUN RGB-D: 55.9% accuracy, surpassing previous methods that relied on complex object detection pipelines.
  • NYUD v2: 67.8% accuracy.

Ablation studies confirmed that the synergy between global layout features and local "theme" features, bolstered by the $L_{sim}$ loss, is critical for achieving these results.

Conclusion

By aligning attention across modalities and adaptively selecting key local features, this framework provides a robust solution for RGB-D scene understanding. While depth sensor noise remains a challenge in background areas, the spatial-attention mechanism effectively mitigates this by focusing on discriminative foreground objects. Future work may extend this approach to outdoor environments or explore deeper modal-consistent representations.

Summary updated

BibTeX
@article{xiong2019rgb,
  author = {Zhitong Xiong and Yuan Yuan and Qi Wang},
  title = {{RGB-D Scene Recognition via Spatial-Related Multi-Modal Feature Learning}},
  journal = {IEEE Access},
  year = {2019},
  doi = {10.1109/access.2019.2932080},
}
Figure