Variational Amodal Object Completion

Huan Ling, David Acuna, Karsten Kreis, Seung Wook Kim, Sanja FidlerNeurIPS 2020 · OpenReview

Introduction

In computer vision, standard "modal" segmentation identifies only the visible parts of an object. However, human perception is naturally amodal—we intuitively understand that a car partially hidden behind a tree is still a complete vehicle, not two disconnected pieces of metal. Teaching AI this capability is challenging because amodal datasets are rare and expensive to create. Asking human annotators to guess and draw the hidden parts of objects is subjective, time-consuming, and often yields messy, unrealistic contours.

To address this, researchers have developed Amodal-VAE, a generative framework that learns to "see behind" occlusions. By learning the prior shape of complete objects, the model can infer the full extent of occluded objects without requiring any manual amodal labels during training.

Methodology: The Amodal-VAE Framework

The core of the proposed method is a Variational Autoencoder (VAE) trained in three distinct stages. This decoupled approach separates the difficulty of learning what an object looks like from learning how it is occluded:

  1. Full-Mask-only Training: The model is initially trained on a dataset of unoccluded objects. It learns a low-dimensional latent space where every point represents a realistic, complete object shape (e.g., a whole car).
  2. Simulated Partial-to-Full-Mask Training: The decoder from the first stage is frozen, ensuring the model can only generate valid shapes. The researchers then simulate occlusions by overlaying other masks onto full masks. A new encoder is trained to map these synthetic partial masks to the latent code of their original, complete versions.
  3. Partial-Mask-only Finetuning: Finally, the model is finetuned on real-world occluded images. Because there are no ground-truth full masks available, the model is optimized to ensure the reconstructed shape perfectly matches the visible "modal" pixels.

To handle objects appearing at various scales and positions, the architecture integrates a Spatial Transformer Network (STN). This allows the VAE to operate on a normalized, cropped view of the object while learning how to accurately paste the completed version back into the original scene.

Key Insights

By freezing the decoder after the first stage, the latent space acts as a strong shape prior. The model is forced to find the closest valid full object in its memory that fits the visible evidence, preventing it from generating unrealistic shapes. Furthermore, unlike deterministic models that output a single best guess, the VAE allows for probabilistic sampling. If a car is heavily obscured, Amodal-VAE can generate multiple plausible hypotheses for its hidden geometry.

Results and Downstream Applications

The researchers evaluated Amodal-VAE on two major street-scene datasets: KINS (which includes amodal annotations for evaluation) and Cityscapes (which lacks amodal labels, testing generalization).

The model achieved state-of-the-art performance on the KINS dataset, outperforming previous de-occlusion methods by 5.66% in mean Intersection over Union (mIOU) for invisible areas.

One of the most striking findings emerged from a blind user study comparing the model's outputs to human annotations. Because humans struggle to draw hidden curves precisely, a generative model trained on perfect unoccluded shapes actually produces more convincing results. Participants preferred the Amodal-VAE completions 46.68% of the time, compared to only 39.50% for the human-annotated ground truth.

Scene Editing Application

This high-quality amodal completion enables powerful downstream applications, such as interactive scene editing. As demonstrated above, by completing occluded objects, users can seamlessly "erase" or "move" foreground vehicles, allowing the system to automatically fill in the background objects with their full, corrected shapes and textures.

Limitations and Future Work

While highly effective for rigid objects like cars and bicycles, the model struggles with highly deformable objects (such as pedestrians in complex poses) because their latent space of possible shapes is vastly more complex. Additionally, while the model does not need amodal labels, it still requires a high-quality dataset of complete masks for each category to learn the initial shape prior. Future research will likely focus on expanding this approach to more cluttered indoor environments and a wider variety of object classes.

Summary updated

Abstract

In images of complex scenes, objects are often occluding each other which makes perception tasks such as object detection and tracking, or robotic control tasks such as planning, challenging. To facilitate downstream tasks, it is thus important to reason about the full extent of objects, i.e., seeing behind occlusion, typically referred to as amodal instance completion. In this paper, we propose a variational generative framework for amodal completion, referred to as AMODAL-VAE, which does not require any amodal labels at training time, as it is able to utilize widely available object instance masks. We showcase our approach on the downstream task of scene editing where the user is presented with interactive tools to complete and erase objects in photographs. Experiments on complex street scenes demonstrate state-of-the-art performance in amodal mask completion and showcase high-quality scene editing results. Interestingly, a user study shows that humans prefer object completions inferred by our model to the human-labeled ones.
BibTeX
@inproceedings{ling2020variational,
  author = {Huan Ling and David Acuna and Karsten Kreis and Seung Wook Kim and Sanja Fidler},
  title = {{Variational Amodal Object Completion}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=CmBNuEuSLzu},
}
Figure