Object-level Scene Deocclusion

Zhengzhe Liu, Qing Liu, Chirui Chang, Jianming Zhang, Daniil Pakhomov, Haitian Zheng, Zhe Lin, Daniel Cohen-Or, Chi-Wing Fu

Abstract

PACO (PArallel visible-to-COmplete) is a generative framework designed to reconstruct the hidden shapes and textures of occluded objects in real-world images. By leveraging diffusion priors and a novel parallel architecture, PACO enables high-fidelity scene deocclusion, facilitating downstream tasks like image editing and single-view 3D reconstruction.

Introduction

In natural images, objects frequently overlap, creating occlusions that hide significant portions of their geometry and appearance. Scene deocclusion aims to predict these hidden "occludee" regions. Unlike standard image inpainting, which often lacks object-specific context or hallucinates entirely new entities, deocclusion requires preserving the identity and structural integrity of existing objects. Previous discriminative methods succeeded in mask prediction but struggled with realistic texture generation. PACO addresses this by providing a generative framework that hallucinates missing parts with high fidelity.

Methodology

The PACO framework utilizes a two-stage training pipeline built upon the Stable Diffusion (SD) architecture.

Parallel Variational Autoencoder (VAE)

The model employs a specialized VAE that processes a stack of objects into a single "full-view" feature map. A decoder then uses a partial query mask to isolate and reconstruct specific objects from this combined latent space. This approach allows the model to handle multiple objects within a single latent representation.

Visualization of object segmentation and masking used as queries in the PACO framework to isolate specific objects.

Visible-to-Complete Latent Generator

A U-Net, fine-tuned from SD, predicts the full-view feature map from visible segments. This process is conditioned on text prompts (e.g., "teddy bear") to resolve ambiguities in object completion, inheriting "common sense" priors from the foundation model.

Layer-wise Inference

To manage complex scenes, PACO groups objects into depth layers using depth estimation. Objects within the same layer are deoccluded simultaneously, preventing feature "bleeding" between adjacent objects during the diffusion pass.

Datasets and Evaluation

The authors introduced the Object Ensemble (OE) Dataset, containing 500,000 synthetic samples derived from 85,000 COCO objects. This dataset enables self-supervised learning by providing ground truths for hidden parts. Evaluation was conducted on COCOA (COCO Amodal), ADE20k, and real-world "in-the-wild" images.

Key Results

PACO demonstrates significant improvements over previous state-of-the-art models like SSSD:

  • Amodal Mask Accuracy: Achieved an IoU of 89.52 on COCOA, compared to SSSD's 87.59.
  • Image Fidelity: Reached a FID score of 13.93, indicating superior reconstruction realism and sharpness.
  • Occlusion Reasoning: Predicts depth order with 90% accuracy.
Deoccluded individual objects from a complex scene, showing completed textures and shapes for each component.

The model's ability to "lift" deoccluded objects into 3D using external tools like DreamCraft3D further highlights its utility for comprehensive scene understanding.

Discussion and Limitations

While PACO sets a new benchmark, it faces challenges with lighting consistency—specifically, shadows from occluders are sometimes treated as permanent textures on the reconstructed object. Additionally, boundary completion for objects partially outside the image frame remains difficult. Future work aims to automate the pipeline further, reducing reliance on manual segmentation and labeling tools.

Summary updated

BibTeX
@inproceedings{liu2024object,
  author = {Zhengzhe Liu and Qing Liu and Chirui Chang and Jianming Zhang and Daniil Pakhomov and Haitian Zheng and Zhe Lin and Daniel Cohen-Or and Chi-Wing Fu},
  title = {{Object-level Scene Deocclusion}},
  booktitle = {SIGGRAPH (Conference Paper Track)},
  year = {2024},
  doi = {10.1145/3641519.3657409},
}
Figure