Time-Conditioned Generative Modeling of Object-Centric Representations for Video Decomposition and Prediction

Chengmin Gao, Bin Li

Introduction

When humans perceive the world, we possess an innate ability for "object constancy." We understand that a coffee mug remains a complete 3D volume even when partially obscured, and we can easily imagine how a scene might look from a different perspective. In computer vision, Object-Centric Learning seeks to replicate this ability by decomposing complex scenes into independent "slots" or representations for each object. However, current models often struggle with two primary challenges: accurately reconstructing occluded parts of objects and predicting novel viewpoints without expensive, manually-labeled camera coordinates.

This paper introduces a novel time-conditioned generative model designed to address these hurdles. By integrating Transformers, sequential Slot Attention, and Gaussian Processes (GPs), the proposed framework learns to decompose videos into individual objects and their respective viewpoints in an entirely unsupervised manner.

The Challenge of Object Constancy and Viewpoint Dependency

Most existing object-centric models fail to maintain a consistent representation of an object when it is partially or fully blocked by another item in a specific frame. Furthermore, predicting how a scene appears from a new angle typically requires "viewpoint annotations"—exact camera coordinates that are rarely available in real-world video data. The authors propose a model that learns the implicit rules of perspective and 3D structure simply by observing a camera move around a static scene, treating the camera's path as a continuous function of time rather than a set of discrete, labeled coordinates.

Proposed Methodology

The architecture is built upon three main pillars designed to disentangle object properties from camera movement:

  1. Disentangled Latent Representations: The model separates the scene into "Object Latents" ($z^{obj}$), which capture 3D properties like shape and texture, and "View Latents" ($z^{view}$), which represent the camera's position. This ensures that object representations remain view-invariant.
  2. Transformer-Based View Inference and Sequential Slot Attention: A Transformer block processes the video sequence to infer the camera's trajectory. These view representations are then fed into a sequential version of Slot Attention, which groups pixels into individual object slots.
  3. Gaussian Processes (GPs) as Priors: To enable novel-view prediction without camera labels, the model uses GPs to model the viewpoint latents over time. By treating the camera's movement as a smooth, continuous function, the model can interpolate or extrapolate to unseen time stamps (and thus unseen viewpoints).
  4. Depth-Based Rendering: To handle occlusions, the model estimates a depth value for each object slot. It employs a "weighted normalization" process, similar to a painter's algorithm, ensuring that objects closer to the camera correctly mask those further away.
The proposed generative architecture, illustrating the integration of Gaussian Processes for view latents and sequential Slot Attention for object-centric representations.

Experimental Evaluation

The model was evaluated on four synthetic multi-view video datasets: CLEVR-SIMPLE, CLEVR-COMPLEX, SHOP-SIMPLEX, and SHOP-COMPLEX. The "SHOP" datasets are particularly challenging as they feature realistic 3D scans of household items with complex textures and irregular shapes.

Key Results

  • Superior Decomposition: On the challenging SHOP-COMPLEX dataset, the model achieved an ARI-O score of 94.9, significantly outperforming other unsupervised models like SIMONe (33.5) and OCLOC (29.3).
  • Handling Occlusion: The model successfully reconstructed objects that were 100% occluded in certain frames by synthesizing information from other viewpoints where the object was visible.
  • Label-Free Prediction: Despite having no access to camera coordinates, the model's novel-view prediction performance was competitive with state-of-the-art models that require explicit viewpoint annotations.
Qualitative results on SHOP-COMPLEX showing (a) RGB frames, (b) segmentation masks, (c) depth-based occlusion reasoning, and (d) individual object reconstructions across observed and predicted time steps.

Discussion and Future Directions

The research demonstrates that it is possible to learn 3D perspective and object-centric decomposition implicitly from video data. By forcing the model to reconstruct the same scene from multiple time-stamped frames, it develops an internal 3D model without ever seeing a coordinate label. The use of Gaussian Processes naturally accounts for the increasing uncertainty as the predicted viewpoint moves further away from the observed frames.

Currently, the model assumes static scenes where only the camera moves. Extending this framework to dynamic scenes with independently moving objects is a key area for future work. Additionally, while the SHOP datasets provide a high level of complexity, testing the model on real-world video data, such as drone or robotic sensor footage, remains the next logical step in validating its robustness.

Summary updated

Abstract

When perceiving the world from multiple viewpoints, humans have the ability to reason about the complete objects in a compositional manner even when an object is completely occluded from certain viewpoints. Meanwhile, humans are able to imagine novel views after observing multiple viewpoints. Recent remarkable advances in multi-view object-centric learning still leaves some unresolved problems: 1) The shapes of partially or completely occluded objects can not be well reconstructed. 2) The novel viewpoint prediction depends on expensive viewpoint annotations rather than implicit rules in view representations. In this paper, we introduce a time-conditioned generative model for videos. To reconstruct the complete shape of an object accurately, we enhance the disentanglement between the latent representations of objects and views, where the latent representations of time-conditioned views are jointly inferred with a Transformer and then are input to a sequential extension of Slot Attention to learn object-centric representations. In addition, Gaussian processes are employed as priors of view latent variables for video generation and novel-view prediction without viewpoint annotations. Experiments on multiple datasets demonstrate that the proposed model can make object-centric video decomposition, reconstruct the complete shapes of occluded objects, and make novel-view predictions.
BibTeX
@inproceedings{gao2023time,
  author = {Chengmin Gao and Bin Li},
  title = {{Time-Conditioned Generative Modeling of Object-Centric Representations for Video Decomposition and Prediction}},
  booktitle = {Conference on Uncertainty in Artificial Intelligence},
  year = {2023},
  url = {https://openreview.net/forum?id=4LA00Cww33},
}
Figure