IAM: Enhancing RGB-D Instance Segmentation with New Benchmarks

Aecheon Jung, Soyun Choi, Junhong Min, Sungeun HongarXiv 2025 · DOI · arXiv

Abstract

While RGB-D data provides rich geometric cues essential for identifying objects in complex indoor environments, research has primarily focused on semantic segmentation. This paper addresses the critical shortage of instance segmentation datasets and the limitations of current multimodal fusion techniques. We introduce three new benchmarks—NYUDv2-IS, SUN-RGBD-IS, and Box-IS—alongside a novel Intra-modal Attention Mix (IAM) module. By prioritizing within-modality feature relationships before fusion, IAM significantly improves object boundary delineation. Our approach outperforms existing state-of-the-art methods, demonstrating consistent gains in segmentation accuracy across diverse indoor scenarios.

Introduction

RGB-D (Red, Green, Blue + Depth) sensors are pivotal for indoor navigation and robotic manipulation, offering geometric information that remains robust under challenging lighting or low-texture conditions. Despite these advantages, the field lacks comprehensive datasets for instance segmentation, which requires distinguishing between individual objects of the same category. Furthermore, existing fusion strategies often suffer from "modality imbalance," where models fail to fully leverage the fine-grained geometric details provided by depth data.

The Intra-modal Attention Mix (IAM) Module

To address these challenges, we propose the IAM module. Unlike traditional inter-modal fusion that focuses on cross-modality interactions, IAM emphasizes enhancing relationships within each modality (intra-modal) before integration.

The module consists of three key components:

  1. Feature Compression: Employs 1x1 convolutions to reduce the dimensionality of concatenated RGB and Depth features, minimizing computational costs.
  2. Mixup-inspired Attention: This mechanism calculates attention scores by summing independent self-attention maps from both modalities. This preserves modality-specific details while facilitating efficient feature integration.
  3. Channel-wise Dynamic Fusion (CDF): CDF uses Global Average Pooling and a sigmoid function to dynamically weight feature channels, ensuring that the most relevant information from both RGB and Depth is propagated through the network.

We integrated IAM into transformer-based architectures like DETR and SOLQ, finding optimal performance when placed between the later blocks of a ResNet50 backbone.

New Benchmarks for RGB-D Instance Segmentation

We introduce three benchmarks to fill the data gap:

  • NYUDv2-IS: A re-annotated version of the NYU Depth V2 dataset focusing on 9 instance categories.
  • SUN-RGBD-IS: A large-scale benchmark with 17 categories and nearly 10,000 images.
  • Box-IS: A specialized dataset for high-precision robotic tasks, featuring complex, overlapping box arrangements captured with an Intel RealSense D455.

Key Insights and Results

A central finding of this work is the role of depth as a primary delineator. While RGB features can become noisy in low-light or cluttered scenes, depth features consistently provide sharp object boundaries.

Visual comparison of feature maps across modalities. Depth features (c) provide significantly clearer object boundaries compared to RGB (b), leading to superior fused RGB-D representations (d).

The IAM module demonstrated significant performance improvements:

  • NYUDv2-IS: Achieved a 6.7% gain in $AP^{seg}$ over early fusion methods.
  • SUN-RGBD-IS: Reached 25.7% $AP^{seg}$ using the SOLQ architecture, significantly outperforming RGB-only baselines.
  • Box-IS: Proved highly effective at separating overlapping objects in industrial settings.

Conclusion

The IAM module and the accompanying benchmarks represent a significant step forward for RGB-D instance segmentation. By focusing on intra-modal attention, we mitigate modality imbalance and better preserve geometric details. Future work will focus on optimizing the module for real-time performance and exploring its generalization to outdoor LiDAR-based datasets.

Summary updated

Abstract

Image segmentation is a vital task for providing human assistance and enhancing autonomy in our daily lives. In particular, RGB-D segmentation-leveraging both visual and depth cues-has attracted increasing attention as it promises richer scene understanding than RGB-only methods. However, most existing efforts have primarily focused on semantic segmentation and thus leave a critical gap. There is a relative scarcity of instance-level RGB-D segmentation datasets, which restricts current methods to broad category distinctions rather than fully capturing the fine-grained details required for recognizing individual objects. To bridge this gap, we introduce three RGB-D instance segmentation benchmarks, distinguished at the instance level. These datasets are versatile, supporting a wide range of applications from indoor navigation to robotic manipulation. In addition, we present an extensive evaluation of various baseline models on these benchmarks. This comprehensive analysis identifies both their strengths and shortcomings, guiding future work toward more robust, generalizable solutions. Finally, we propose a simple yet effective method for RGB-D data integration. Extensive evaluations affirm the effectiveness of our approach, offering a robust framework for advancing toward more nuanced scene understanding.
BibTeX
@misc{jung2025iam,
  author = {Aecheon Jung and Soyun Choi and Junhong Min and Sungeun Hong},
  title = {{IAM: Enhancing RGB-D Instance Segmentation with New Benchmarks}},
  year = {2025},
  eprint = {2501.01685},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2501.01685},
}
Figure