Learning Semantics-aware Distance Map with Semantics Layering Network for Amodal Instance Segmentation

Ziheng Zhang, Anpei Chen, Ling Xie, Jingyi Yu, Shenghua Gao

TL;DR

Researchers from ShanghaiTech University have developed a new way for AI to "see" occluded objects by introducing the semantics-aware distance map (sem-dist map). By treating occlusion as a series of depth layers rather than just a flat mask, their Semantics Layering Network (SLN) achieves state-of-the-art performance in amodal instance segmentation—the ability to perceive the full shape of an object even when it's partially hidden.

What Problem Does This Paper Address?

Standard computer vision models excel at modal segmentation: identifying the visible pixels of an object. However, humans possess amodal perception, the cognitive ability to understand that an object (like a car parked behind a tree) exists as a complete entity even if parts of it are invisible.

Existing AI approaches to amodal segmentation usually try to regress a binary mask for the hidden parts. This is difficult because it ignores two critical pieces of information:

  1. Depth Order: Which object is in front of which?
  2. Occlusion Complexity: Predicting a part hidden behind one object is easier than predicting a part buried behind three.

The authors argue that current models treat occlusion as "noise" to be overcome, rather than a structured relationship between objects that can be explicitly modeled.

Methods

The paper introduces two major innovations: a new data representation and a specific network architecture.

1. The Sem-dist Map

Instead of a simple 0-or-1 mask, the authors propose the semantics-aware distance map. This is a "level-set" representation where:

  • The Integer Part represents the visibility level (how many objects are occluding that specific pixel). Level 0 is visible; Level 1 is hidden behind one object, and so on.
  • The Fractional Part represents the confidence of the object's existence at that pixel.

This single map unifiedly describes the visible (modal) mask, the full (amodal) mask, and the relative depth order of all overlapping objects.

2. Semantics Layering Network (SLN)

To estimate these maps, the authors built the SLN, a two-stage CNN consisting of:

  • Encoder (ENC): A dilated ResNet-50 that extracts high-level features.
  • Global Layering Module (GLM): This predicts a "global layering map" for the entire image, essentially counting how many objects overlap at every pixel.
  • Region Proposal Network (RPN): Generates amodal bounding box proposals.
  • Instance Layering Module (ILM): Takes the cropped features and the global layering info to produce the final, detailed sem-dist map for each specific object.

Datasets

The model was evaluated on two primary benchmarks:

  • COCOA (COCO Amodal): A dataset of 5,000 images with amodal annotations for both "things" (cars, people) and "stuff" (sky, grass).
  • D2SA (Densely Segmented Supermarket Amodal): A newer dataset containing 5,600 images of supermarket items, often in highly cluttered, overlapping configurations.

Key Insights and Ideas

  • Occlusion as a Feature, Not a Bug: By explicitly predicting the "layer" of occlusion, the network learns the spatial hierarchy of the scene.
  • Thresholding for Free: Because of the sem-dist map's structure, you can extract the visible mask by looking only at values between 0 and 1, and the full amodal mask by taking the fractional part of the entire map.
  • Relative Depth via Subtraction: By comparing the integer values of two overlapping sem-dist maps, the model can mathematically determine which object is in front without needing a separate "depth" branch.
  • Global-to-Instance Flow: Predicting the global overlap count first (via the GLM) provides a powerful "prior" that helps the instance-level branch understand the context of occlusion.

Key Results

The SLN outperformed previous state-of-the-art methods (like AmodalMask and ARCNN) across multiple metrics:

  • Amodal Segmentation: On the COCOA dataset, SLN achieved a Mean Average Precision (mAP) of 8.4, significantly higher than AmodalMask's 5.74.
  • Occlusion Handling: The model showed particular strength in "heavy occlusion" scenarios, where objects are mostly hidden.
  • Depth Ordering: The model achieved 85.4% accuracy in predicting the pairwise depth order of objects, outperforming specialized baselines like OrderNet (79.1%).
  • Qualitative Success: Visual results show the model successfully "completing" complex shapes, such as the legs of a horse hidden behind a fence or a sofa partially obscured by a table.

Limitations and Future Work

The authors identify the Region Proposal Network (RPN) as the current bottleneck. Because the RPN relies on bounding boxes, it struggles to propose accurate "amodal" boxes when only a tiny fraction of an object is visible.

Future work may involve moving toward proposal-free architectures or pixel-level proposal modules to better handle cases where objects are almost entirely buried behind others. Additionally, extending this to video to leverage temporal consistency in occlusion layers is a logical next step.

Summary updated

Abstract

In this work, we demonstrate yet another approach to tackle the amodal segmentation problem. Specifically, we first introduce a new representation, namely a semantics-aware distance map (sem-dist map), to serve as our target for amodal segmentation instead of the commonly used masks and heatmaps. The sem-dist map is a kind of level-set representation, of which the different regions of an object are placed into different levels on the map according to their visibility. It is a natural extension of masks and heatmaps, where modal, amodal segmentation, as well as depth order information, are all well-described. Then we also introduce a novel convolutional neural network (CNN) architecture, which we refer to as semantic layering network, to estimate sem-dist maps layer by layer, from the global-level to the instance-level, for all objects in an image. Extensive experiments on the COCOA and D2SA datasets have demonstrated that our framework can predict amodal segmentation, occlusion, and depth order with state-of-the-art performance.
BibTeX
@inproceedings{zhang2019learning,
  author = {Ziheng Zhang and Anpei Chen and Ling Xie and Jingyi Yu and Shenghua Gao},
  title = {{Learning Semantics-aware Distance Map with Semantics Layering Network for Amodal Instance Segmentation}},
  booktitle = {ACM Multimedia},
  year = {2019},
  url = {https://openreview.net/forum?id=5M4Mc3cbwHT},
}

Loading related content...

Figure