Exploring the Benefits of Depth Information in Object Pixel Masking (Student Abstract)

Anish Kachinthaya, Yi Ding, Tobias Höllerer

Abstract

State-of-the-art instance segmentation models like Mask R-CNN often struggle with occlusions and overlapping subjects. This study investigates the integration of depth information via simple thresholding techniques to improve masking accuracy in complex scenes. Our results indicate that depth data can recover pixels missed by RGB-only models, particularly in occluded regions, though challenges remain in distinguishing subjects from the ground.

Introduction

Pixel masking, or instance segmentation, is a fundamental task in computer vision, essential for applications ranging from sports analytics to autonomous surveillance. While Mask R-CNN has established itself as a robust baseline, its reliance on RGB features makes it susceptible to errors when visual boundaries are ambiguous due to occlusion or similar textures. This research explores whether depth information, providing physical distance data, can mitigate these limitations.

Methodology

The proposed approach utilizes a hybrid pipeline combining deep learning with classical computer vision:

  1. Detection: A pre-trained Mask R-CNN identifies initial bounding boxes for individuals.
  2. Depth Processing: Depth matrices are extracted for each box using an Intel RealSense D415 camera.
  3. Thresholding: Otsu’s method is applied to the depth histogram to automatically calculate an optimal threshold separating the subject from the background.
  4. Optimization: Through empirical tuning, masking pixels below the 85th percentile of the threshold depth was found to yield the most accurate results.

Experimental Results

The method was evaluated on a custom dataset of 101 RGB-D images featuring challenging occlusion scenarios.

Quantitative Analysis

A confusion matrix was used to compare the pixel-wise agreement between the standard Mask R-CNN and the proposed depth-based thresholder. The two methods achieved a total overlap of 68.83%. Notably, the depth method identified 6.99% of pixels that were omitted by Mask R-CNN. However, it also exhibited a 24.17% "over-masking" rate, primarily due to the difficulty of separating a subject's feet from the floor.

Confusion matrix comparing pixel-wise overlap between the Depth Thresholder and Mask R-CNN.

Qualitative Comparison

Visual inspection highlights the complementary strengths of the two approaches. While Mask R-CNN provides finer detail for features like hair, the depth thresholder is more effective at capturing the global extent of occluded individuals in the background that the RGB model may ignore.

Side-by-side comparison of masks generated by the Depth Thresholder (left) and Mask R-CNN (right).

Discussion and Conclusion

Depth information proves to be a robust feature, largely invariant to lighting and color. Its explainability offers a significant advantage over "black box" neural networks, as thresholds are based on physical distance. The primary drawback is the "ground challenge," where similar depth values between the subject and the floor lead to over-masking. Future research will focus on sensor fusion to combine the visual precision of RGB with the structural reliability of depth data into a single, unified model.

Summary updated

Abstract

In this paper, we look at how depth data can benefit existing object masking methods applied in occluded scenes. Masking the pixel locations of objects within scenes helps computers get a spatial awareness of where objects are within images. The current state-of-the-art algorithm for masking objects in images is Mask R-CNN, which builds on the Faster R-CNN network to mask object pixels rather than just detecting their bounding boxes. This paper examines the weaknesses Mask R-CNN has in masking people when they are occluded in a frame. It then looks at how depth data gathered from an RGB-D sensor can be used. We provide a case study to show how simply applying thresholding methods on the depth information can aid in distinguishing occluded persons. The intention of our research is to examine how features from depth data can benefit object pixel masking methods in an explainable manner, especially in complex scenes with multiple objects.
BibTeX
@inproceedings{kachinthaya2020exploring,
  author = {Anish Kachinthaya and Yi Ding and Tobias Höllerer},
  title = {{Exploring the Benefits of Depth Information in Object Pixel Masking (Student Abstract)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i10.7189},
}
Figure