Abstract
PACO (PArallel visible-to-COmplete) is a generative framework designed to reconstruct the hidden shapes and textures of occluded objects in real-world images. By leveraging diffusion priors and a novel parallel architecture, PACO enables high-fidelity scene deocclusion, facilitating downstream tasks like image editing and single-view 3D reconstruction.
Introduction
In natural images, objects frequently overlap, creating occlusions that hide significant portions of their geometry and appearance. Scene deocclusion aims to predict these hidden "occludee" regions. Unlike standard image inpainting, which often lacks object-specific context or hallucinates entirely new entities, deocclusion requires preserving the identity and structural integrity of existing objects. Previous discriminative methods succeeded in mask prediction but struggled with realistic texture generation. PACO addresses this by providing a generative framework that hallucinates missing parts with high fidelity.
Methodology
The PACO framework utilizes a two-stage training pipeline built upon the Stable Diffusion (SD) architecture.
Parallel Variational Autoencoder (VAE)
The model employs a specialized VAE that processes a stack of objects into a single "full-view" feature map. A decoder then uses a partial query mask to isolate and reconstruct specific objects from this combined latent space. This approach allows the model to handle multiple objects within a single latent representation.
Visible-to-Complete Latent Generator
A U-Net, fine-tuned from SD, predicts the full-view feature map from visible segments. This process is conditioned on text prompts (e.g., "teddy bear") to resolve ambiguities in object completion, inheriting "common sense" priors from the foundation model.
Layer-wise Inference
To manage complex scenes, PACO groups objects into depth layers using depth estimation. Objects within the same layer are deoccluded simultaneously, preventing feature "bleeding" between adjacent objects during the diffusion pass.
Datasets and Evaluation
The authors introduced the Object Ensemble (OE) Dataset, containing 500,000 synthetic samples derived from 85,000 COCO objects. This dataset enables self-supervised learning by providing ground truths for hidden parts. Evaluation was conducted on COCOA (COCO Amodal), ADE20k, and real-world "in-the-wild" images.
Key Results
PACO demonstrates significant improvements over previous state-of-the-art models like SSSD:
- Amodal Mask Accuracy: Achieved an IoU of 89.52 on COCOA, compared to SSSD's 87.59.
- Image Fidelity: Reached a FID score of 13.93, indicating superior reconstruction realism and sharpness.
- Occlusion Reasoning: Predicts depth order with 90% accuracy.
The model's ability to "lift" deoccluded objects into 3D using external tools like DreamCraft3D further highlights its utility for comprehensive scene understanding.
Discussion and Limitations
While PACO sets a new benchmark, it faces challenges with lighting consistency—specifically, shadows from occluders are sometimes treated as permanent textures on the reconstructed object. Additionally, boundary completion for objects partially outside the image frame remains difficult. Future work aims to automate the pipeline further, reducing reliance on manual segmentation and labeling tools.
Summary updated