TL;DR
This paper introduces an automated pipeline that leverages 3D scene data to generate authentic, objective amodal segmentation ground truths for real-world images, producing the new MP3D-Amodal dataset. Additionally, the authors propose two novel architectures—including a highly effective one-stage model utilizing Stable Diffusion features—that perform class-agnostic amodal completion "in the wild" without requiring prior knowledge of the occluding object.
What Problem Does This Paper Address?
Humans possess an innate ability for amodal perception—we easily understand that an object continues to exist in its full shape even when it is partially blocked by something else. Standard instance segmentation models (like Mask R-CNN or SAM), however, only perform modal segmentation, mapping out exclusively the visible pixels.
Training models to perceive the hidden parts of objects (amodal segmentation) is incredibly challenging due to a lack of clean data. Existing real-image datasets rely on human annotators to subjectively "guess" and draw the hidden contours, which introduces human bias and error. Synthetic datasets provide perfect ground truth shapes but lack real-world visual complexity. Furthermore, most existing amodal completion algorithms require the user to explicitly provide a mask of the occluder (the blocking object) at inference time, making them impractical for real-world deployment where occluders are often unannotated or unknown.
Methods
The authors tackle these challenges by introducing two architecture variants that eliminate the need for an explicit occluder mask during inference:
- OccAmodal (Two-Stage Model): This network first uses an Occluder Predictor (built on a lightweight ResNet and U-Net) to explicitly estimate the occluder mask from the image and the target object's visible mask. An Amodal Predictor then takes this predicted occluder and the resulting occlusion boundary to complete the full object shape.
- SDAmodal (One-Stage Model): Instead of explicitly predicting what is blocking the object, this architecture leverages the rich structural and geometric priors embedded within a pre-trained, frozen Stable Diffusion model. By adding noise to the latent features and extracting multi-scale features from the diffusion model's U-Net decoder at a specific time step (step 181), the model implicitly captures deep occlusion relationships. These features are then combined with the visible mask features to decode the final completed object shape.
Datasets
- MP3D-Amodal (New): The primary data contribution of this paper. It consists of 12,724 authentic amodal masks across 10,883 real indoor images derived from the Matterport3D dataset. Because Matterport3D contains both real images and aligned 3D meshes of rooms, the authors created true ground truths by projecting individual 3D object meshes into the 2D camera view to get the unoccluded shape.
- COCOA / COCOA-cls: Existing real-image benchmarks used for training and comparative evaluation, where the hidden shapes were manually guessed by human annotators.
Key Insights and Ideas
- Stable Diffusion as a Geometric Prior: Generative text-to-image models like Stable Diffusion do not just understand textures; they develop an internal representation of 3D space, object boundaries, and layering. Tapping into these features yields massive performance gains for spatial tasks like amodal completion.
- Authentic Real-World Ground Truth: By utilizing 3D sensor data (scans/meshes) mapped to 2D images, the authors bypassed the traditional trade-off between synthetic data (perfect masks, fake images) and human-annotated data (imperfect masks, real images).
- Implicit vs. Explicit Occlusion Reasoning: Moving from explicit two-stage occluder prediction to implicit feature extraction (SDAmodal) proves that generative features can "hallucinate" hidden shapes more effectively than models forced to explicitly define what is blocking them.
Key Results
- State-of-the-Art Performance: On the COCOA dataset, SDAmodal achieved a state-of-the-art mean Intersection-over-Union (mIOU) of 90.7, and an mIOU-inv (which measures accuracy strictly within the hidden regions) of 71.6. Crucially, it outperformed previous methods even when those older methods were given the ground-truth occluder masks.
- Zero-Shot Generalization: When trained purely on COCOA and evaluated zero-shot on the new MP3D-Amodal dataset, SDAmodal achieved 76.4 mIOU, demonstrating excellent class-agnostic and domain-agnostic generalization.
- Feature Comparison: Replacing Stable Diffusion features with other popular self-supervised representations caused performance drops (DINOv2 dropped mIOU to 90.3; OpenCLIP dropped it to 88.3), validating that Stable Diffusion features are uniquely well-suited for occlusion tasks.
Limitations and Future Work
- Mesh Quality Dependency: The dataset generation pipeline is bound by the quality of the initial 3D scans. Matterport3D meshes can occasionally be noisy or incomplete, which forced the authors to include a manual human-verification step to filter out bad proposals.
- Extreme Occlusions: While the models handle a wide range of occlusions, predicting shapes when an object is almost entirely hidden (e.g., >90% occluded) remains an open problem where models must rely more on pure hallucination than visual context.
- Scaling Up: The authors note that this 3D-to-2D projection pipeline can be extended to other large-scale RGB-D datasets, such as ScanNet, to further scale up real-world amodal training data.
Summary updated