CMANet: Cross-Modality Attention Network for Indoor-Scene Semantic Segmentation

Longze Zhu, Zhizhong Kang, Mei Zhou, Xi Yang, Zhen Wang, Zhen Cao, Chenming Ye2022 · DOI

TL;DR

This paper introduces CMANet, a novel deep learning architecture for indoor semantic segmentation that effectively fuses RGB and HHA (depth-derived) image data. By employing a Cross-Modality Refine Gate (CMRG) with self-attention mechanisms, CMANet significantly improves segmentation accuracy and robustness, outperforming existing methods on benchmark datasets.

What Problem Does This Paper Address?

Indoor semantic segmentation is crucial for applications like indoor navigation, robotics, and augmented reality. However, it's a challenging task due to the complex textures, occlusions, and lighting variations common in indoor environments. While RGB images capture appearance, they struggle with depth and structural ambiguity. Depth images, on the other hand, provide geometric information but are often noisy and lack rich texture details. HHA images, derived from depth, offer a more robust representation of geometric information than raw depth. The core challenge lies in effectively fusing these complementary modalities (RGB and HHA) to leverage their strengths and overcome their individual weaknesses for accurate pixel-wise scene understanding. Existing methods often struggle with feature fusion, leading to suboptimal performance, especially when dealing with noisy or incomplete data.

Methods

CMANet adopts an encoder-decoder architecture. The encoder features two parallel branches, one for RGB and one for HHA images, both utilizing ResNet-50 as a backbone to extract latent features. The novelty lies in the Cross-Modality Refine Gate (CMRG), a custom attention mechanism designed to fuse features from both modalities.

The CMRG consists of two main parts:

  • Cross-Modality Channel Attention: This module aggregates spatial information from both RGB and HHA features (using average and max pooling) and then uses shared MLPs to learn modality-specific attention maps. This helps filter out noise and highlight important features within each modality.
  • Cross-Modality Spatial Attention: This module further refines the channel-attended features by learning spatial attention maps. It focuses on identifying and emphasizing relevant regions across both modalities, strengthening the understanding of spatial relationships and object boundaries.

Crucially, the CMRG operates on features from both modalities simultaneously, unlike standard attention mechanisms that might operate on each modality independently. This cross-modal interaction is key to its effectiveness.

Additionally, CMANet incorporates:

  • Bi-Directional Multi-Step Propagation (BMP): This technique helps propagate refined features between the encoder branches, reducing model complexity and improving efficiency by averaging rather than simply adding features.
  • Pyramid Supervision: This strategy applies loss functions at multiple resolution levels within the decoder, helping to mitigate gradient vanishing and improve performance across different scales.

Datasets

The paper evaluates CMANet on two widely used indoor RGB-D semantic segmentation datasets:

  • NYUDv2: Contains 1449 indoor RGB-D images with dense pixel-wise annotations, split into 795 training and 654 testing images. It uses a 40-category setting.
  • SUN RGB-D: A larger dataset with 10,335 indoor RGB-D images across 37 categories, including images from NYUDv2 and other sources. It's split into 5285 training and 5050 testing images.

Key Insights and Ideas

  • Cross-Modality Attention is Key: The core insight is that a dedicated attention mechanism that explicitly models interactions between RGB and HHA features (CMRG) is more effective than simply concatenating or adding features. This allows the network to learn which features from each modality are most relevant to the other.
  • HHA Data is Crucial for Geometry: The paper demonstrates that HHA images, derived from depth, provide complementary geometric information that significantly boosts segmentation accuracy, especially for object shapes and spatial relationships that are ambiguous in RGB alone.
  • Multi-Stage Fusion Matters: Applying the CMRG at multiple stages of the encoder allows for progressive refinement of cross-modality features, leading to better overall performance.
  • Attention for Robustness: The attention mechanisms within CMRG help filter out noise and irrelevant information, making the model more robust to variations in lighting and sensor data quality.

Key Results

CMANet achieves state-of-the-art performance on both NYUDv2 and SUN RGB-D datasets.

  • NYUDv2: CMANet (with ResNet-50 backbone) achieves 74.2% pixel accuracy, 60.6% mean accuracy, and 47.6% mean IoU. This is a notable improvement over methods like RefineNet-101 and LSD-GF.
  • SUN RGB-D: CMANet achieves 47.2% mean IoU, outperforming several other state-of-the-art methods.
  • Ablation Studies: The results show that the CMRG is the most significant contributor to performance improvement, followed by skip connections and pyramid supervision. Removing CMRGs at earlier stages has a more detrimental effect than removing them at later stages.
  • Model Efficiency: CMANet offers a good balance between accuracy and efficiency, achieving comparable or better performance than existing methods with reduced parameters and FLOPs.

Limitations and Future Work

  • Backbone Dependency: While CMANet performs well with ResNet-50, the authors note that using a more powerful backbone could further improve results.
  • Dataset Scale: The authors acknowledge that the NYUDv2 dataset is relatively small.
  • Future Work: The authors plan to focus on increasing network efficiency (reducing time and space complexity) and exploring semi-supervised or weakly-supervised learning strategies due to dataset limitations and potential labeling inaccuracies.

Summary updated

BibTeX
@misc{zhu2022cmanet,
  author = {Longze Zhu and Zhizhong Kang and Mei Zhou and Xi Yang and Zhen Wang and Zhen Cao and Chenming Ye},
  title = {{CMANet: Cross-Modality Attention Network for Indoor-Scene Semantic Segmentation}},
  year = {2022},
  doi = {10.3390/s22218520},
}
Figure