Introduction
When humans perceive the world, we possess an innate ability for "object constancy." We understand that a coffee mug remains a complete 3D volume even when partially obscured, and we can easily imagine how a scene might look from a different perspective. In computer vision, Object-Centric Learning seeks to replicate this ability by decomposing complex scenes into independent "slots" or representations for each object. However, current models often struggle with two primary challenges: accurately reconstructing occluded parts of objects and predicting novel viewpoints without expensive, manually-labeled camera coordinates.
This paper introduces a novel time-conditioned generative model designed to address these hurdles. By integrating Transformers, sequential Slot Attention, and Gaussian Processes (GPs), the proposed framework learns to decompose videos into individual objects and their respective viewpoints in an entirely unsupervised manner.
The Challenge of Object Constancy and Viewpoint Dependency
Most existing object-centric models fail to maintain a consistent representation of an object when it is partially or fully blocked by another item in a specific frame. Furthermore, predicting how a scene appears from a new angle typically requires "viewpoint annotations"—exact camera coordinates that are rarely available in real-world video data. The authors propose a model that learns the implicit rules of perspective and 3D structure simply by observing a camera move around a static scene, treating the camera's path as a continuous function of time rather than a set of discrete, labeled coordinates.
Proposed Methodology
The architecture is built upon three main pillars designed to disentangle object properties from camera movement:
- Disentangled Latent Representations: The model separates the scene into "Object Latents" ($z^{obj}$), which capture 3D properties like shape and texture, and "View Latents" ($z^{view}$), which represent the camera's position. This ensures that object representations remain view-invariant.
- Transformer-Based View Inference and Sequential Slot Attention: A Transformer block processes the video sequence to infer the camera's trajectory. These view representations are then fed into a sequential version of Slot Attention, which groups pixels into individual object slots.
- Gaussian Processes (GPs) as Priors: To enable novel-view prediction without camera labels, the model uses GPs to model the viewpoint latents over time. By treating the camera's movement as a smooth, continuous function, the model can interpolate or extrapolate to unseen time stamps (and thus unseen viewpoints).
- Depth-Based Rendering: To handle occlusions, the model estimates a depth value for each object slot. It employs a "weighted normalization" process, similar to a painter's algorithm, ensuring that objects closer to the camera correctly mask those further away.
Experimental Evaluation
The model was evaluated on four synthetic multi-view video datasets: CLEVR-SIMPLE, CLEVR-COMPLEX, SHOP-SIMPLEX, and SHOP-COMPLEX. The "SHOP" datasets are particularly challenging as they feature realistic 3D scans of household items with complex textures and irregular shapes.
Key Results
- Superior Decomposition: On the challenging SHOP-COMPLEX dataset, the model achieved an ARI-O score of 94.9, significantly outperforming other unsupervised models like SIMONe (33.5) and OCLOC (29.3).
- Handling Occlusion: The model successfully reconstructed objects that were 100% occluded in certain frames by synthesizing information from other viewpoints where the object was visible.
- Label-Free Prediction: Despite having no access to camera coordinates, the model's novel-view prediction performance was competitive with state-of-the-art models that require explicit viewpoint annotations.
Discussion and Future Directions
The research demonstrates that it is possible to learn 3D perspective and object-centric decomposition implicitly from video data. By forcing the model to reconstruct the same scene from multiple time-stamped frames, it develops an internal 3D model without ever seeing a coordinate label. The use of Gaussian Processes naturally accounts for the increasing uncertainty as the predicted viewpoint moves further away from the observed frames.
Currently, the model assumes static scenes where only the camera moves. Extending this framework to dynamic scenes with independently moving objects is a key area for future work. Additionally, while the SHOP datasets provide a high level of complexity, testing the model on real-world video data, such as drone or robotic sensor footage, remains the next logical step in validating its robustness.
Summary updated