Abstract
In this work, we demonstrate yet another approach to tackle the amodal segmentation problem. Specifically, we first introduce a new representation, namely a semantics-aware distance map (sem-dist map), to serve as our target for amodal segmentation instead of the commonly used masks and heatmaps. The sem-dist map is a kind of level-set representation, of which the different regions of an object are placed into different levels on the map according to their visibility. It is a natural extension of masks and heatmaps, where modal, amodal segmentation, as well as depth order information, are all well-described. Then we also introduce a novel convolutional neural network (CNN) architecture, which we refer to as semantic layering network, to estimate sem-dist maps layer by layer, from the global-level to the instance-level, for all objects in an image. Extensive experiments on the COCOA and D2SA datasets have demonstrated that our framework can predict amodal segmentation, occlusion and depth order with state-of-the-art performance.
Introduction
Standard computer vision models excel at modal segmentation, which involves identifying the visible pixels of an object. However, humans possess amodal perception, the cognitive ability to understand that an object exists as a complete entity even if parts of it are invisible, such as a car parked behind a tree.
Existing AI approaches to amodal segmentation typically attempt to regress a binary mask for the hidden parts. This approach is limited because it often ignores critical contextual information like depth order and occlusion complexity. The authors argue that occlusion should be treated as a structured relationship between objects that can be explicitly modeled rather than mere noise.
Methods
The proposed framework introduces two major innovations: a new data representation called the semantics-aware distance map and a specific network architecture, the Semantics Layering Network (SLN).
The Sem-dist Map
Instead of a binary 0-or-1 mask, the semantics-aware distance map (sem-dist map) uses a level-set representation. The integer part of the map represents the visibility level (the number of objects occluding a specific pixel), where Level 0 is visible and Level 1 is hidden behind one object. The fractional part represents the confidence of the object's existence at that pixel. This unified representation describes the visible (modal) mask, the full (amodal) mask, and the relative depth order of overlapping objects.
Semantics Layering Network (SLN)
The SLN is a two-stage CNN consisting of:
- Encoder (ENC): A dilated ResNet-50 for high-level feature extraction.
- Global Layering Module (GLM): Predicts a global layering map to count overlapping objects at every pixel.
- Region Proposal Network (RPN): Generates amodal bounding box proposals.
- Instance Layering Module (ILM): Produces the final, detailed sem-dist map for each specific object using cropped features and global layering information.
Results
The SLN was evaluated on the COCOA (COCO Amodal) and D2SA (Densely Segmented Supermarket Amodal) datasets. It outperformed previous state-of-the-art methods like AmodalMask and ARCNN across multiple metrics:
- Amodal Segmentation: On COCOA, SLN achieved a Mean Average Precision (mAP) of 8.4, compared to AmodalMask's 5.74.
- Occlusion Handling: The model demonstrated significant strength in "heavy occlusion" scenarios.
- Depth Ordering: The model achieved 85.4% accuracy in predicting pairwise depth order, outperforming specialized baselines like OrderNet (79.1%).
Conclusion
By explicitly predicting the "layer" of occlusion, the SLN learns the spatial hierarchy of a scene. While the current bottleneck is the Region Proposal Network's reliance on bounding boxes for heavily occluded objects, future work may involve proposal-free architectures or leveraging temporal consistency in video to further improve amodal perception.
Summary updated