TL;DR
Researchers from ShanghaiTech University have developed a new way for AI to "see" occluded objects by introducing the semantics-aware distance map (sem-dist map). By treating occlusion as a series of depth layers rather than just a flat mask, their Semantics Layering Network (SLN) achieves state-of-the-art performance in amodal instance segmentation—the ability to perceive the full shape of an object even when it's partially hidden.
What Problem Does This Paper Address?
Standard computer vision models excel at modal segmentation: identifying the visible pixels of an object. However, humans possess amodal perception, the cognitive ability to understand that an object (like a car parked behind a tree) exists as a complete entity even if parts of it are invisible.
Existing AI approaches to amodal segmentation usually try to regress a binary mask for the hidden parts. This is difficult because it ignores two critical pieces of information:
- Depth Order: Which object is in front of which?
- Occlusion Complexity: Predicting a part hidden behind one object is easier than predicting a part buried behind three.
The authors argue that current models treat occlusion as "noise" to be overcome, rather than a structured relationship between objects that can be explicitly modeled.
Methods
The paper introduces two major innovations: a new data representation and a specific network architecture.
1. The Sem-dist Map
Instead of a simple 0-or-1 mask, the authors propose the semantics-aware distance map. This is a "level-set" representation where:
- The Integer Part represents the visibility level (how many objects are occluding that specific pixel). Level 0 is visible; Level 1 is hidden behind one object, and so on.
- The Fractional Part represents the confidence of the object's existence at that pixel.
This single map unifiedly describes the visible (modal) mask, the full (amodal) mask, and the relative depth order of all overlapping objects.
2. Semantics Layering Network (SLN)
To estimate these maps, the authors built the SLN, a two-stage CNN consisting of:
- Encoder (ENC): A dilated ResNet-50 that extracts high-level features.
- Global Layering Module (GLM): This predicts a "global layering map" for the entire image, essentially counting how many objects overlap at every pixel.
- Region Proposal Network (RPN): Generates amodal bounding box proposals.
- Instance Layering Module (ILM): Takes the cropped features and the global layering info to produce the final, detailed sem-dist map for each specific object.
Datasets
The model was evaluated on two primary benchmarks:
- COCOA (COCO Amodal): A dataset of 5,000 images with amodal annotations for both "things" (cars, people) and "stuff" (sky, grass).
- D2SA (Densely Segmented Supermarket Amodal): A newer dataset containing 5,600 images of supermarket items, often in highly cluttered, overlapping configurations.
Key Insights and Ideas
- Occlusion as a Feature, Not a Bug: By explicitly predicting the "layer" of occlusion, the network learns the spatial hierarchy of the scene.
- Thresholding for Free: Because of the sem-dist map's structure, you can extract the visible mask by looking only at values between 0 and 1, and the full amodal mask by taking the fractional part of the entire map.
- Relative Depth via Subtraction: By comparing the integer values of two overlapping sem-dist maps, the model can mathematically determine which object is in front without needing a separate "depth" branch.
- Global-to-Instance Flow: Predicting the global overlap count first (via the GLM) provides a powerful "prior" that helps the instance-level branch understand the context of occlusion.
Key Results
The SLN outperformed previous state-of-the-art methods (like AmodalMask and ARCNN) across multiple metrics:
- Amodal Segmentation: On the COCOA dataset, SLN achieved a Mean Average Precision (mAP) of 8.4, significantly higher than AmodalMask's 5.74.
- Occlusion Handling: The model showed particular strength in "heavy occlusion" scenarios, where objects are mostly hidden.
- Depth Ordering: The model achieved 85.4% accuracy in predicting the pairwise depth order of objects, outperforming specialized baselines like OrderNet (79.1%).
- Qualitative Success: Visual results show the model successfully "completing" complex shapes, such as the legs of a horse hidden behind a fence or a sofa partially obscured by a table.
Limitations and Future Work
The authors identify the Region Proposal Network (RPN) as the current bottleneck. Because the RPN relies on bounding boxes, it struggles to propose accurate "amodal" boxes when only a tiny fraction of an object is visible.
Future work may involve moving toward proposal-free architectures or pixel-level proposal modules to better handle cases where objects are almost entirely buried behind others. Additionally, extending this to video to leverage temporal consistency in occlusion layers is a logical next step.
Summary updated