Abstract
This paper presents a generative model designed to decompose video sequences into individual 3D object representations and predict novel viewpoints without requiring camera pose annotations. By modeling viewpoint transitions as a Gaussian Process over time and utilizing a sequential Slot Attention mechanism, the model achieves robust amodal perception, accurately reconstructing objects even when fully occluded. Evaluations on synthetic datasets demonstrate that the model significantly outperforms existing unsupervised methods in segmentation and amodal completion.
Introduction
A hallmark of human visual intelligence is amodal perception—the capacity to understand that objects maintain their integrity even when partially or fully obscured by other elements in a scene. Furthermore, humans possess the ability to imagine a scene from novel perspectives after minimal observation. Replicating these abilities in artificial systems remains a significant challenge in object-centric learning.
Existing models often struggle with two key issues: (1) inaccurate reconstruction of occluded regions, leading to artifacts or "hallucinations," and (2) a dependency on ground-truth camera coordinates (e.g., azimuth and elevation) for novel view synthesis. This research addresses these limitations by introducing a time-conditioned generative model that learns implicit geometric rules from temporal data alone, eliminating the need for expensive camera pose annotations.
Methodology
The core of the proposed approach is the disentanglement of object identity (content) from the camera viewpoint (pose).
Viewpoint Modeling with Gaussian Processes
The model treats the camera viewpoint as a latent variable $z_t^{\text{view}}$ that evolves continuously over time. By employing a Gaussian Process (GP) as a prior, the model captures the temporal correlation between frames. This allows the system to "interpolate" or "extrapolate" viewpoints for any given timestamp $t$, effectively discovering the underlying 3D structure of the scene without explicit geometric supervision.
Sequential Slot Attention and Amodal Reconstruction
To identify and represent individual objects, the model utilizes an extension of Slot Attention. Features from multiple video frames are distilled into a set of "slots," where each slot represents a view-invariant description of a specific object.
The model further incorporates a depth-ordering module. By learning to predict the relative distance of objects, the generator can render complete object shapes for background elements before overlaying foreground objects. This explicit modeling of occlusion enables high-fidelity amodal reconstruction.
Experimental Results
The model was evaluated on four synthetic multi-view video datasets: CLEVR-SIMPLE, CLEVR-COMPLEX, SHOP-SIMPLE, and SHOP-COMPLEX. These datasets simulate orbital camera movement around static scenes containing multiple objects.
Decomposition and Segmentation
On the challenging SHOP-COMPLEX dataset, the proposed model achieved an ARI-O score of 94.9, substantially higher than baseline methods such as SIMONe (33.5) and OCLOC (29.3). This indicates a superior ability to consistently identify objects across varying viewpoints.
Amodal Completion and Viewpoint Prediction
The model demonstrates a remarkable ability to reconstruct objects that are completely hidden in certain frames. As shown in the qualitative results, the model successfully predicts the full geometry of occluded items. Quantitatively, it achieved an Intersection over Union (IoU) for complete shapes of 66.2 on SHOP-C, compared to 21.5 for OCLOC.
Furthermore, the model's ability to predict novel views using only timestamps is competitive with supervised methods like MulMON, which require full camera pose information.
Conclusion
This work introduces a novel framework for unsupervised object-centric learning that leverages temporal continuity to discover 3D scene structure. By disentangling viewpoint from object identity and utilizing GP priors, the model achieves state-of-the-art performance in video decomposition and amodal perception. Future research will focus on extending this approach to dynamic scenes with moving objects and improving scalability for real-world video applications.
Summary updated