Learning Semantics-aware Distance Map with Semantics Layering Network for Amodal Instance Segmentation

Ziheng Zhang, Anpei Chen, Ling Xie, Jingyi Yu, Shenghua Gao

Abstract

In this work, we demonstrate yet another approach to tackle the amodal segmentation problem. Specifically, we first introduce a new representation, namely a semantics-aware distance map (sem-dist map), to serve as our target for amodal segmentation instead of the commonly used masks and heatmaps. The sem-dist map is a kind of level-set representation, of which the different regions of an object are placed into different levels on the map according to their visibility. It is a natural extension of masks and heatmaps, where modal, amodal segmentation, as well as depth order information, are all well-described. Then we also introduce a novel convolutional neural network (CNN) architecture, which we refer to as semantic layering network, to estimate sem-dist maps layer by layer, from the global-level to the instance-level, for all objects in an image. Extensive experiments on the COCOA and D2SA datasets have demonstrated that our framework can predict amodal segmentation, occlusion and depth order with state-of-the-art performance.

Introduction

Standard computer vision models excel at modal segmentation, which involves identifying the visible pixels of an object. However, humans possess amodal perception, the cognitive ability to understand that an object exists as a complete entity even if parts of it are invisible, such as a car parked behind a tree.

The SLN framework estimates sem-dist maps layer by layer, progressing from global-level to instance-level features.

Existing AI approaches to amodal segmentation typically attempt to regress a binary mask for the hidden parts. This approach is limited because it often ignores critical contextual information like depth order and occlusion complexity. The authors argue that occlusion should be treated as a structured relationship between objects that can be explicitly modeled rather than mere noise.

Methods

The proposed framework introduces two major innovations: a new data representation called the semantics-aware distance map and a specific network architecture, the Semantics Layering Network (SLN).

The Sem-dist Map

Instead of a binary 0-or-1 mask, the semantics-aware distance map (sem-dist map) uses a level-set representation. The integer part of the map represents the visibility level (the number of objects occluding a specific pixel), where Level 0 is visible and Level 1 is hidden behind one object. The fractional part represents the confidence of the object's existence at that pixel. This unified representation describes the visible (modal) mask, the full (amodal) mask, and the relative depth order of overlapping objects.

Overview of the sem-dist map representation, showing how modal and amodal masks are derived and how relative distance determines depth order.

Semantics Layering Network (SLN)

The SLN is a two-stage CNN consisting of:

  • Encoder (ENC): A dilated ResNet-50 for high-level feature extraction.
  • Global Layering Module (GLM): Predicts a global layering map to count overlapping objects at every pixel.
  • Region Proposal Network (RPN): Generates amodal bounding box proposals.
  • Instance Layering Module (ILM): Produces the final, detailed sem-dist map for each specific object using cropped features and global layering information.

Results

The SLN was evaluated on the COCOA (COCO Amodal) and D2SA (Densely Segmented Supermarket Amodal) datasets. It outperformed previous state-of-the-art methods like AmodalMask and ARCNN across multiple metrics:

  • Amodal Segmentation: On COCOA, SLN achieved a Mean Average Precision (mAP) of 8.4, compared to AmodalMask's 5.74.
  • Occlusion Handling: The model demonstrated significant strength in "heavy occlusion" scenarios.
  • Depth Ordering: The model achieved 85.4% accuracy in predicting pairwise depth order, outperforming specialized baselines like OrderNet (79.1%).

Conclusion

By explicitly predicting the "layer" of occlusion, the SLN learns the spatial hierarchy of a scene. While the current bottleneck is the Region Proposal Network's reliance on bounding boxes for heavily occluded objects, future work may involve proposal-free architectures or leveraging temporal consistency in video to further improve amodal perception.

Summary updated

Abstract

In this work, we demonstrate yet another approach to tackle the amodal segmentation problem. Specifically, we first introduce a new representation, namely a semantics-aware distance map (sem-dist map), to serve as our target for amodal segmentation instead of the commonly used masks and heatmaps. The sem-dist map is a kind of level-set representation, of which the different regions of an object are placed into different levels on the map according to their visibility. It is a natural extension of masks and heatmaps, where modal, amodal segmentation, as well as depth order information, are all well-described. Then we also introduce a novel convolutional neural network (CNN) architecture, which we refer to as semantic layering network, to estimate sem-dist maps layer by layer, from the global-level to the instance-level, for all objects in an image. Extensive experiments on the COCOA and D2SA datasets have demonstrated that our framework can predict amodal segmentation, occlusion and depth order with state-of-the-art performance.
BibTeX
@inproceedings{zhang2019learning,
  author = {Ziheng Zhang and Anpei Chen and Ling Xie and Jingyi Yu and Shenghua Gao},
  title = {{Learning Semantics-aware Distance Map with Semantics Layering Network for Amodal Instance Segmentation}},
  booktitle = {ACM Multimedia},
  year = {2019},
  url = {https://openreview.net/forum?id=5M4Mc3cbwHT},
}
Figure