Introduction
Scene recognition is a fundamental challenge in computer vision that differs significantly from standard object recognition. While identifying a "dog" is often object-centric, identifying a "living room" requires understanding the global layout and the spatial arrangement of various objects within an environment. This complexity is further compounded when using RGB-D data, which combines color (RGB) and geometric (Depth) information.
The Challenge of Multi-Modal Scene Understanding
Current approaches to RGB-D scene recognition face two primary hurdles:
- Multi-modal Fusion: Effectively combining color data with geometric depth information to create a unified representation.
- Local Feature Extraction: Capturing discriminative local details (like a stove in a kitchen) without the computational overhead of dense sampling or the need for expensive, manually-labeled object-level annotations.
Standard Convolutional Neural Networks (CNNs) often struggle with the high spatial variability of scenes when they rely solely on global features. The authors sought a way to adaptively find the most important parts of a scene to boost classification accuracy.
Proposed Methodology
The authors introduce a two-branch CNN architecture designed to learn both global scene layouts and "key" local features simultaneously.
Key Feature Selection (KFS) Module
The core innovation is the KFS module, which employs a spatial attention mechanism based on non-local networks. It adaptively identifies high-response regions in feature maps—representing scene "themes"—and selects the top K local feature vectors. This allows for weakly supervised localization of important scene elements without requiring explicit object labels.
Novel Loss Functions
To optimize the multi-modal learning process, the framework introduces two specific loss functions:
- Spatial-Attention Similarity Loss ($L_{sim}$): This loss forces the attention masks of the RGB and Depth branches to align. Since both sensors observe the same physical space, this alignment helps the depth branch ignore sensor noise by "following the lead" of the cleaner RGB signal.
- Triplet Correlation Loss: This ensures that local features from images within the same category are highly correlated, while those from different categories remain distinct.
Experimental Results
The model was evaluated on two major indoor scene benchmarks: SUN RGB-D (10,355 images across 19 categories) and NYU Depth v2 (1,449 images across 10 categories).
The proposed method achieved state-of-the-art (SOTA) performance at the time of publication:
- SUN RGB-D: 55.9% accuracy, surpassing previous methods that relied on complex object detection pipelines.
- NYUD v2: 67.8% accuracy.
Ablation studies confirmed that the synergy between global layout features and local "theme" features, bolstered by the $L_{sim}$ loss, is critical for achieving these results.
Conclusion
By aligning attention across modalities and adaptively selecting key local features, this framework provides a robust solution for RGB-D scene understanding. While depth sensor noise remains a challenge in background areas, the spatial-attention mechanism effectively mitigates this by focusing on discriminative foreground objects. Future work may extend this approach to outdoor environments or explore deeper modal-consistent representations.
Summary updated