Amodal Intra-class Instance Segmentation: Synthetic Datasets and Benchmark

Jiayang Ao, Qiuhong Ke, Krista A. EhingerWACV 2024 · OpenReview

TL;DR

This paper tackles the challenging problem of amodal instance segmentation, specifically when objects of the same class heavily occlude each other. To address the lack of suitable datasets, they introduce two large-scale synthetic datasets and a novel point-supervised method incorporating "layer priors" that significantly improves performance on these difficult occlusion scenarios.

What Problem Does This Paper Address?

Amodal perception, the ability to understand an object's complete shape even when parts are hidden, is crucial for tasks like robotic grasping and autonomous driving. Amodal instance segmentation aims to achieve this computationally by identifying both visible and occluded parts of objects. However, a major hurdle is dealing with intra-class occlusion, where objects of the same type (e.g., multiple pears) overlap significantly. Existing datasets often lack the scale and detailed annotations needed to explicitly model these dense, same-class occlusions, making it hard for models to generalize. This paper aims to fill this gap by providing new datasets and a method specifically designed for this challenging scenario.

Methods

The core of the proposed method, named PLIn (Pointly-supervised scheme with Layer priors for amodal Intra-class instance segmentation), is built upon a few key innovations:

  • Point-Supervised Learning: Instead of requiring full pixel-level annotations for amodal masks (which are expensive and prone to annotator bias), PLIn uses a much weaker supervision signal: just 10 random points per object. This significantly reduces annotation effort while still providing a strong learning signal.
  • Layer Priors: To explicitly handle intra-class occlusion, PLIn treats overlapping objects as belonging to different "layers." This allows the model to disentangle instances that are visually similar but spatially distinct due to occlusion. The model learns to predict a {category, layer} pair for each instance, effectively creating distinct "classes" for each layer of a given object type.
  • Cascade Strategy: To further improve detection robustness in dense occlusion, PLIn employs a 3-stage cascade training strategy. This means the output of one detection stage is used as input for the next, progressively refining bounding box predictions across different Intersection over Union (IoU) thresholds.
  • Subdivision Mask Rendering: After predicting points, a subdivision mask rendering algorithm is used to refine the final amodal instance masks.

Datasets

The paper introduces two novel synthetic datasets:

  • Intra-AFruit: This is the primary dataset, containing over 267K images with dense intra-class occlusion scenarios focused on fruits and vegetables. It includes detailed annotations for visible, invisible, and amodal masks, amodal bounding boxes, and dual order relations (occlusion and layer order). It's split into 210K training and 45K testing images.
  • Inter-AMix: A dataset designed to test generalization, featuring inter-class occlusion scenarios.
  • ACom (Amodal Common Objects): This dataset diversifies object categories beyond fruits and vegetables, including common objects like bottles, donuts, etc., to test broader applicability.

They also collected a small real-world dataset of intra-class occlusion scenes for evaluation.

Key Insights and Ideas

  • The Power of Layering: Explicitly modeling objects as belonging to different layers, even within the same class, is a highly effective way to disentangle heavily occluded instances. This is a significant departure from traditional methods that might struggle to differentiate visually similar, overlapping objects.
  • Weak Supervision is Viable: The success of point-based supervision for amodal tasks, especially with the proposed layer priors, demonstrates that expensive full-mask annotations might not always be necessary, opening doors for more scalable amodal perception research.
  • Intra-class Occlusion is a Unique Challenge: The paper highlights that standard NMS (Non-Maximum Suppression) techniques, common in object detection, can be detrimental in dense intra-class occlusion scenarios, often erroneously removing highly occluded instances. Their layer-based approach directly addresses this limitation.

Key Results

  • PLIn, using only 10-point supervision, achieves state-of-the-art (SOTA) performance on the Intra-AFruit test set, outperforming fully supervised methods by a significant margin (e.g., >9.63% APMask and >6.97% APBox).
  • The proposed method shows strong generalization capabilities on the Inter-AMix (inter-class occlusion) and real-world datasets, outperforming most other methods.
  • Ablation studies confirm the effectiveness of both Layer Priors and the Cascade strategy, with Layer Priors improving amodal mask prediction and Cascade strategy boosting bounding box accuracy. Test-Time Augmentation (TTA) also provides a modest performance boost.
  • Qualitative results show PLIn successfully detecting and segmenting instances that other methods miss or misinterpret due to heavy occlusion.

Limitations and Future Work

  • Layer Order Prediction Difficulty: While Layer Priors improve performance, the paper notes that predicting deeper layers (more occluded) is inherently harder, and performance degrades as layers increase. This is partly attributed to fewer training instances in deeper layers.
  • Computational Cost of TTA: Test-Time Augmentation, while beneficial, is noted as being time-consuming, which might limit its real-time applicability.
  • Future Work: The authors suggest their datasets and method provide a strong benchmark for amodal instance segmentation with intra-class occlusion. Future work could explore further improvements in layer order prediction and potentially extend the point-based supervision to other amodal completion tasks.

Summary updated

Abstract

Images of realistic scenes often contain intra-class objects that are heavily occluded from each other, making the amodal perception task that requires parsing the occluded parts of the objects challenging. Although important for downstream tasks such as robotic grasping systems, the lack of large-scale amodal datasets with detailed annotations makes it difficult to model intra-class occlusions explicitly. This paper introduces two new amodal datasets for image amodal completion tasks, which contain a total of over 267K images of intra-class occlusion scenarios, annotated with multiple masks, amodal bounding boxes, dual order relations and full appearance for instances and background. We also present a point-supervised scheme with layer priors for amodal instance segmentation specifically designed for intra-class occlusion scenarios 1 . Experiments show that our weakly supervised approach outperforms the SOTA fully supervised methods, while our layer priors design exhibits remarkable performance improvements in the case of intra-class occlusion in both synthetic and real images.
BibTeX
@inproceedings{ao2024amodal,
  author = {Jiayang Ao and Qiuhong Ke and Krista A. Ehinger},
  title = {{Amodal Intra-class Instance Segmentation: Synthetic Datasets and Benchmark}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
  url = {https://openreview.net/forum?id=O20yipIw3w},
}
Figure