Time-Conditioned Generative Modeling of Object-Centric Representations for Video Decomposition and Prediction

Chengmin Gao, Bin Li

Abstract

This paper presents a generative model designed to decompose video sequences into individual 3D object representations and predict novel viewpoints without requiring camera pose annotations. By modeling viewpoint transitions as a Gaussian Process over time and utilizing a sequential Slot Attention mechanism, the model achieves robust amodal perception, accurately reconstructing objects even when fully occluded. Evaluations on synthetic datasets demonstrate that the model significantly outperforms existing unsupervised methods in segmentation and amodal completion.

Introduction

A hallmark of human visual intelligence is amodal perception—the capacity to understand that objects maintain their integrity even when partially or fully obscured by other elements in a scene. Furthermore, humans possess the ability to imagine a scene from novel perspectives after minimal observation. Replicating these abilities in artificial systems remains a significant challenge in object-centric learning.

Existing models often struggle with two key issues: (1) inaccurate reconstruction of occluded regions, leading to artifacts or "hallucinations," and (2) a dependency on ground-truth camera coordinates (e.g., azimuth and elevation) for novel view synthesis. This research addresses these limitations by introducing a time-conditioned generative model that learns implicit geometric rules from temporal data alone, eliminating the need for expensive camera pose annotations.

Methodology

The core of the proposed approach is the disentanglement of object identity (content) from the camera viewpoint (pose).

Viewpoint Modeling with Gaussian Processes

The model treats the camera viewpoint as a latent variable $z_t^{\text{view}}$ that evolves continuously over time. By employing a Gaussian Process (GP) as a prior, the model captures the temporal correlation between frames. This allows the system to "interpolate" or "extrapolate" viewpoints for any given timestamp $t$, effectively discovering the underlying 3D structure of the scene without explicit geometric supervision.

Sequential Slot Attention and Amodal Reconstruction

To identify and represent individual objects, the model utilizes an extension of Slot Attention. Features from multiple video frames are distilled into a set of "slots," where each slot represents a view-invariant description of a specific object.

The inference architecture employs a Transformer-based viewpoint encoder and a CNN-based object encoder to map video sequences into disentangled latent representations.

The model further incorporates a depth-ordering module. By learning to predict the relative distance of objects, the generator can render complete object shapes for background elements before overlaying foreground objects. This explicit modeling of occlusion enables high-fidelity amodal reconstruction.

Experimental Results

The model was evaluated on four synthetic multi-view video datasets: CLEVR-SIMPLE, CLEVR-COMPLEX, SHOP-SIMPLE, and SHOP-COMPLEX. These datasets simulate orbital camera movement around static scenes containing multiple objects.

Decomposition and Segmentation

On the challenging SHOP-COMPLEX dataset, the proposed model achieved an ARI-O score of 94.9, substantially higher than baseline methods such as SIMONe (33.5) and OCLOC (29.3). This indicates a superior ability to consistently identify objects across varying viewpoints.

Amodal Completion and Viewpoint Prediction

The model demonstrates a remarkable ability to reconstruct objects that are completely hidden in certain frames. As shown in the qualitative results, the model successfully predicts the full geometry of occluded items. Quantitatively, it achieved an Intersection over Union (IoU) for complete shapes of 66.2 on SHOP-C, compared to 21.5 for OCLOC.

Qualitative results illustrating (a) frame reconstruction, (b) object segmentation, (c) predicted overlaps, and (d) complete amodal segmentation for both observed and predicted timestamps.

Furthermore, the model's ability to predict novel views using only timestamps is competitive with supervised methods like MulMON, which require full camera pose information.

Conclusion

This work introduces a novel framework for unsupervised object-centric learning that leverages temporal continuity to discover 3D scene structure. By disentangling viewpoint from object identity and utilizing GP priors, the model achieves state-of-the-art performance in video decomposition and amodal perception. Future research will focus on extending this approach to dynamic scenes with moving objects and improving scalability for real-world video applications.

Summary updated

BibTeX
@inproceedings{gao2023time,
  author = {Chengmin Gao and Bin Li},
  title = {{Time-Conditioned Generative Modeling of Object-Centric Representations for Video Decomposition and Prediction}},
  booktitle = {Conference on Uncertainty in Artificial Intelligence},
  year = {2023},
  url = {https://openreview.net/forum?id=4LA00Cww33},
}
Figure