Introduction
Standard semantic segmentation operates on a "winner-takes-all" principle: each pixel in a 2D image is assigned a single label corresponding to the topmost visible object. While effective for simple classification, this approach creates a significant blind spot for autonomous systems. For instance, a robot needs to know that a floor continues beneath a table to plan a path, and a self-driving car must reason about the road surface hidden behind a parked vehicle. This paper introduces Grouped Semantic Segmentation (GSS), a framework that enables neural networks to perform "amodal perception"—the ability to see behind visible objects and predict semantic labels for occluded surfaces.
Methodology: Grouped Semantic Segmentation
Instead of a single softmax output over all possible classes, GSS partitions object categories into $M$ distinct functional groups (e.g., "Background," "Chair-like," "Table-like"). This restructuring allows the model to assign multiple labels to a single pixel coordinate—one from each group.
Key Technical Components
- Multi-Label Output: By predicting a label for every group at every pixel, the model can simultaneously identify a visible foreground object (e.g., a chair) and an occluded background surface (e.g., the floor).
- The "Void" Category: To prevent the model from hallucinating objects where none exist, each group includes a "void" label. This indicates the absence of that specific category group at a given coordinate.
- Weighted Loss Function: The authors employ a modified cross-entropy loss where visible objects receive full weight, while occluded regions and "void" labels are assigned a lower weight ($\lambda = 0.1$). This encourages the network to prioritize visible accuracy while still learning the underlying scene structure.
- Architecture: Notably, the researchers utilize a standard U-Net architecture. Their findings suggest that the ability to reason about occlusions is a matter of problem framing and supervision rather than requiring increased model complexity or specialized 3D architectures.
Datasets and Training
Since standard datasets lack ground truth for hidden surfaces, the authors developed augmented versions of existing datasets:
- SUNCG (Augmented): Using a synthetic 3D indoor dataset, they rendered scenes multiple times, removing foreground objects in successive passes to reveal the ground truth of occluded surfaces.
- Cityscapes (Augmented): For real-world driving scenarios, they superimposed mobile objects (pedestrians, cars) onto existing scenes, using the original labels as the known "occluded" ground truth.
Results and Insights
The GSS method was compared against Direct Semantic Segmentation (DSS), the standard single-label approach.
- Performance Stability: GSS maintained high accuracy on visible pixels (0.833 PA vs. 0.850 for DSS), ensuring that the secondary task of occlusion prediction does not degrade primary performance.
- Occlusion Accuracy: GSS significantly outperformed the baseline in predicting "present" pixels (visible + hidden). In the SUNCG dataset, GSS achieved a Present Pixel Accuracy (PApres) of 0.900, compared to 0.715 for the baseline.
- Latent Capacity: A key insight of the study is that standard CNNs already possess the latent capacity to predict occluded regions. The model learns statistical regularities—such as the fact that floors are continuous and walls meet at corners—to "fill in" the blanks effectively.
Conclusion
Grouped Semantic Segmentation demonstrates that amodal perception can be achieved without specialized, high-complexity models. By simply restructuring the output space and providing appropriate supervision, standard architectures like U-Net can learn to reason about the hidden structure of the world. Future work remains to be done in reducing the dependency on synthetic data and automating the grouping of semantic categories.
Summary updated