Abstract
State-of-the-art instance segmentation models like Mask R-CNN often struggle with occlusions and overlapping subjects. This study investigates the integration of depth information via simple thresholding techniques to improve masking accuracy in complex scenes. Our results indicate that depth data can recover pixels missed by RGB-only models, particularly in occluded regions, though challenges remain in distinguishing subjects from the ground.
Introduction
Pixel masking, or instance segmentation, is a fundamental task in computer vision, essential for applications ranging from sports analytics to autonomous surveillance. While Mask R-CNN has established itself as a robust baseline, its reliance on RGB features makes it susceptible to errors when visual boundaries are ambiguous due to occlusion or similar textures. This research explores whether depth information, providing physical distance data, can mitigate these limitations.
Methodology
The proposed approach utilizes a hybrid pipeline combining deep learning with classical computer vision:
- Detection: A pre-trained Mask R-CNN identifies initial bounding boxes for individuals.
- Depth Processing: Depth matrices are extracted for each box using an Intel RealSense D415 camera.
- Thresholding: Otsu’s method is applied to the depth histogram to automatically calculate an optimal threshold separating the subject from the background.
- Optimization: Through empirical tuning, masking pixels below the 85th percentile of the threshold depth was found to yield the most accurate results.
Experimental Results
The method was evaluated on a custom dataset of 101 RGB-D images featuring challenging occlusion scenarios.
Quantitative Analysis
A confusion matrix was used to compare the pixel-wise agreement between the standard Mask R-CNN and the proposed depth-based thresholder. The two methods achieved a total overlap of 68.83%. Notably, the depth method identified 6.99% of pixels that were omitted by Mask R-CNN. However, it also exhibited a 24.17% "over-masking" rate, primarily due to the difficulty of separating a subject's feet from the floor.
Qualitative Comparison
Visual inspection highlights the complementary strengths of the two approaches. While Mask R-CNN provides finer detail for features like hair, the depth thresholder is more effective at capturing the global extent of occluded individuals in the background that the RGB model may ignore.
Discussion and Conclusion
Depth information proves to be a robust feature, largely invariant to lighting and color. Its explainability offers a significant advantage over "black box" neural networks, as thresholds are based on physical distance. The primary drawback is the "ground challenge," where similar depth values between the subject and the floor lead to over-masking. Future research will focus on sensor fusion to combine the visual precision of RGB with the structural reliability of depth data into a single, unified model.
Summary updated