Seeing Behind Things: Extending Semantic Segmentation to Occluded Regions

Pulak Purkait, Christopher Zach, Ian D. Reid

Introduction

Standard semantic segmentation operates on a "winner-takes-all" principle: each pixel in a 2D image is assigned a single label corresponding to the topmost visible object. While effective for simple classification, this approach creates a significant blind spot for autonomous systems. For instance, a robot needs to know that a floor continues beneath a table to plan a path, and a self-driving car must reason about the road surface hidden behind a parked vehicle. This paper introduces Grouped Semantic Segmentation (GSS), a framework that enables neural networks to perform "amodal perception"—the ability to see behind visible objects and predict semantic labels for occluded surfaces.

Methodology: Grouped Semantic Segmentation

Instead of a single softmax output over all possible classes, GSS partitions object categories into $M$ distinct functional groups (e.g., "Background," "Chair-like," "Table-like"). This restructuring allows the model to assign multiple labels to a single pixel coordinate—one from each group.

Comparison between (a) Direct Semantic Segmentation, which predicts only visible surfaces, and (b) the proposed Grouped Semantic Segmentation (GSS), which decomposes the scene into functional layers to capture occluded regions.

Key Technical Components

  • Multi-Label Output: By predicting a label for every group at every pixel, the model can simultaneously identify a visible foreground object (e.g., a chair) and an occluded background surface (e.g., the floor).
  • The "Void" Category: To prevent the model from hallucinating objects where none exist, each group includes a "void" label. This indicates the absence of that specific category group at a given coordinate.
  • Weighted Loss Function: The authors employ a modified cross-entropy loss where visible objects receive full weight, while occluded regions and "void" labels are assigned a lower weight ($\lambda = 0.1$). This encourages the network to prioritize visible accuracy while still learning the underlying scene structure.
  • Architecture: Notably, the researchers utilize a standard U-Net architecture. Their findings suggest that the ability to reason about occlusions is a matter of problem framing and supervision rather than requiring increased model complexity or specialized 3D architectures.

Datasets and Training

Since standard datasets lack ground truth for hidden surfaces, the authors developed augmented versions of existing datasets:

  1. SUNCG (Augmented): Using a synthetic 3D indoor dataset, they rendered scenes multiple times, removing foreground objects in successive passes to reveal the ground truth of occluded surfaces.
  2. Cityscapes (Augmented): For real-world driving scenarios, they superimposed mobile objects (pedestrians, cars) onto existing scenes, using the original labels as the known "occluded" ground truth.

Results and Insights

The GSS method was compared against Direct Semantic Segmentation (DSS), the standard single-label approach.

  • Performance Stability: GSS maintained high accuracy on visible pixels (0.833 PA vs. 0.850 for DSS), ensuring that the secondary task of occlusion prediction does not degrade primary performance.
  • Occlusion Accuracy: GSS significantly outperformed the baseline in predicting "present" pixels (visible + hidden). In the SUNCG dataset, GSS achieved a Present Pixel Accuracy (PApres) of 0.900, compared to 0.715 for the baseline.
  • Latent Capacity: A key insight of the study is that standard CNNs already possess the latent capacity to predict occluded regions. The model learns statistical regularities—such as the fact that floors are continuous and walls meet at corners—to "fill in" the blanks effectively.

Conclusion

Grouped Semantic Segmentation demonstrates that amodal perception can be achieved without specialized, high-complexity models. By simply restructuring the output space and providing appropriate supervision, standard architectures like U-Net can learn to reason about the hidden structure of the world. Future work remains to be done in reducing the dependency on synthetic data and automating the grouping of semantic categories.

Summary updated

Abstract

Semantic segmentation and instance level segmentation made substantial progress in recent years due to the emergence of deep neural networks (DNNs). A number of deep architectures with Convolution Neural Networks (CNNs) were proposed that surpass the traditional machine learning approaches for segmentation by a large margin. These architectures predict the directly observable semantic category of each pixel by usually optimizing a cross-entropy loss. In this work we push the limit of semantic segmentation towards predicting semantic labels of directly visible as well as occluded objects or objects parts, where the network's input is a single depth image. We group the semantic categories into one background and multiple foreground object groups, and we propose a modification of the standard cross-entropy loss to cope with the settings. In our experiments we demonstrate that a CNN trained by minimizing the proposed loss is able to predict semantic categories for visible and occluded object parts without requiring to increase the network size (compared to a standard segmentation task). The results are validated on a newly generated dataset (augmented from SUNCG) dataset.
BibTeX
@inproceedings{purkait2019seeing,
  author = {Pulak Purkait and Christopher Zach and Ian D. Reid},
  title = {{Seeing Behind Things: Extending Semantic Segmentation to Occluded Regions}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2019},
  doi = {10.1109/iros40897.2019.8967582},
}
Figure