Abstract
Amodal Instance Segmentation (AIS) is a challenging computer vision task that requires predicting the full shape of objects, including their occluded parts. Traditional single-view approaches often struggle with the inherent ambiguity of hidden regions. This paper introduces MUVA, a large-scale multi-view AIS benchmark, and MASFormer, a novel transformer-based architecture designed to aggregate information across multiple perspectives. Our results demonstrate that multi-view analysis significantly reduces ambiguity, nearly doubling performance over single-view baselines and showing strong generalization to real-world shopping scenarios.
Introduction
In real-world environments, objects are rarely fully visible. Amodal Instance Segmentation (AIS) seeks to go beyond standard segmentation by inferring the complete geometry of an object, even when it is partially obscured by other items. This capability is crucial for applications such as robotic manipulation, autonomous driving, and automated retail checkout systems.
Historically, AIS research has focused on single-view images. However, a single perspective is often insufficient to determine the true shape of an occluded object. Humans naturally overcome this limitation by moving around an object to gain more information. This paper proposes a shift toward Multi-view Amodal Instance Segmentation (MAIS), providing both a massive synthetic dataset and a specialized model to leverage multi-perspective consistency.
The Ill-Posed Nature of Single-View AIS
AIS is fundamentally an "ill-posed" problem when restricted to a single viewpoint. Because the hidden part of an object is not observed, there are theoretically infinite possible shapes that could complete the object.
As illustrated in Figure 1, a single input image (a) can lead to multiple plausible candidate masks. Without additional information, a model must essentially "guess" the occluded structure. By contrast, a multi-view approach (b) allows the system to observe the object from different angles, effectively removing the guesswork and allowing for a prediction that aligns with the ground truth.
The MUVA Dataset
A major hurdle in AIS research is the difficulty of obtaining accurate ground truth for occluded regions. Human annotators are prone to inconsistency when guessing hidden shapes. To solve this, the authors introduced MUVA (MUlti-View AIS), a synthetic dataset generated using Unity3D.
MUVA consists of 26,406 high-resolution images and nearly 200,000 instances, utilizing 1,801 distinct 3D models of common shopping items. Because the dataset is synthetic, the exact 3D geometry and position of every object are known, allowing for the generation of 100% accurate amodal masks. This "perfect" ground truth is essential for training models to understand the geometric logic of occlusion.
MASFormer: Multi-view Aggregation Transformer
To process multi-view data effectively, the authors developed MASFormer. The architecture consists of two primary stages:
- Feature Extraction Stage (FES): A backbone network and Transformer decoder extract features for every instance across all available views.
- Multi-view Aggregation Stage (MAS): This stage uses two parallel attention mechanisms to fuse information:
- View-level Aggregation: Learns spatial relationships between different objects within the same view to understand the local context.
- Instance-level Aggregation: Learns the relationship of the same object across different views. This allows the model to use information from a clear viewpoint to "fill in" the blanks for a view where the object is heavily occluded.
Results and Generalization
The shift to multi-view analysis yielded dramatic improvements in performance. On the MUVA benchmark, MASFormer achieved 39.4 AP using six views, compared to 23.1 AP for the previous single-view state-of-the-art method, BCNet.
Beyond synthetic benchmarks, the authors tested the model's ability to generalize to real-world data using the D2SA shopping dataset. When MASFormer was trained on a combination of real-world and synthetic MUVA data, its performance on the real-world test set increased from 63.5 AP to 68.4 AP. This demonstrates that the geometric principles learned from the synthetic MUVA dataset are highly relevant to real-world applications like automated checkout.
Conclusion
The MUVA benchmark and MASFormer model establish that multi-view perspectives are a powerful tool for solving the inherent ambiguities of amodal segmentation. By providing a large-scale dataset with perfect ground truth and a novel aggregation architecture, the authors have set a new state-of-the-art for the field. Future research will focus on expanding these methods to more diverse environments beyond the shopping scenario and improving the computational efficiency of multi-view Transformer models.
Summary updated