Abstract
While RGB-D data provides rich geometric cues essential for identifying objects in complex indoor environments, research has primarily focused on semantic segmentation. This paper addresses the critical shortage of instance segmentation datasets and the limitations of current multimodal fusion techniques. We introduce three new benchmarks—NYUDv2-IS, SUN-RGBD-IS, and Box-IS—alongside a novel Intra-modal Attention Mix (IAM) module. By prioritizing within-modality feature relationships before fusion, IAM significantly improves object boundary delineation. Our approach outperforms existing state-of-the-art methods, demonstrating consistent gains in segmentation accuracy across diverse indoor scenarios.
Introduction
RGB-D (Red, Green, Blue + Depth) sensors are pivotal for indoor navigation and robotic manipulation, offering geometric information that remains robust under challenging lighting or low-texture conditions. Despite these advantages, the field lacks comprehensive datasets for instance segmentation, which requires distinguishing between individual objects of the same category. Furthermore, existing fusion strategies often suffer from "modality imbalance," where models fail to fully leverage the fine-grained geometric details provided by depth data.
The Intra-modal Attention Mix (IAM) Module
To address these challenges, we propose the IAM module. Unlike traditional inter-modal fusion that focuses on cross-modality interactions, IAM emphasizes enhancing relationships within each modality (intra-modal) before integration.
The module consists of three key components:
- Feature Compression: Employs 1x1 convolutions to reduce the dimensionality of concatenated RGB and Depth features, minimizing computational costs.
- Mixup-inspired Attention: This mechanism calculates attention scores by summing independent self-attention maps from both modalities. This preserves modality-specific details while facilitating efficient feature integration.
- Channel-wise Dynamic Fusion (CDF): CDF uses Global Average Pooling and a sigmoid function to dynamically weight feature channels, ensuring that the most relevant information from both RGB and Depth is propagated through the network.
We integrated IAM into transformer-based architectures like DETR and SOLQ, finding optimal performance when placed between the later blocks of a ResNet50 backbone.
New Benchmarks for RGB-D Instance Segmentation
We introduce three benchmarks to fill the data gap:
- NYUDv2-IS: A re-annotated version of the NYU Depth V2 dataset focusing on 9 instance categories.
- SUN-RGBD-IS: A large-scale benchmark with 17 categories and nearly 10,000 images.
- Box-IS: A specialized dataset for high-precision robotic tasks, featuring complex, overlapping box arrangements captured with an Intel RealSense D455.
Key Insights and Results
A central finding of this work is the role of depth as a primary delineator. While RGB features can become noisy in low-light or cluttered scenes, depth features consistently provide sharp object boundaries.
The IAM module demonstrated significant performance improvements:
- NYUDv2-IS: Achieved a 6.7% gain in $AP^{seg}$ over early fusion methods.
- SUN-RGBD-IS: Reached 25.7% $AP^{seg}$ using the SOLQ architecture, significantly outperforming RGB-only baselines.
- Box-IS: Proved highly effective at separating overlapping objects in industrial settings.
Conclusion
The IAM module and the accompanying benchmarks represent a significant step forward for RGB-D instance segmentation. By focusing on intra-modal attention, we mitigate modality imbalance and better preserve geometric details. Future work will focus on optimizing the module for real-time performance and exploring its generalization to outdoor LiDAR-based datasets.
Summary updated