Object Centric Video Modeling with Slot Attention

This direction is merged into another direction: object-centric representation (object-centric latent representation). Its last version stays readable and citable here.

Version 1 of 116 papers16 conceptsfrozen 2026-10-04

Most video models treat a scene as a dense grid of pixels. While effective for short-term textures, this approach struggles with the fundamental nature of our world: it is composed of discrete, persisting entities. When a car drives behind a tree, a pixel-level model sees a "melting" of colors; a human sees one object temporarily occluding another.

To bridge this gap, researchers are developing Object-Centric Video Models. These architectures decompose video frames into a set of "slots"—discrete latent variables that each represent a single object. By treating objects as the fundamental units of computation, these models achieve better long-term consistency, physical reasoning, and data efficiency.

Objects as the Words of Video

In natural language processing, we don't model text as a sequence of ink pixels; we model it as a sequence of words. Yi-Fu Wu, Jaesik Yoon, and Sungjin Ahn proposed a similar shift for video in Generative Video Transformer: Can Objects be the Words?.

Their Object-Centric Video Transformer (OCVT) uses unsupervised discovery to break a scene into structured latent tokens. Each token encodes specific attributes like appearance, position, and depth. This decomposition provides a powerful inductive bias: instead of the transformer's self-attention mechanism scaling quadratically with the number of pixels, it scales with the number of objects. This makes long-term reasoning computationally feasible and physically grounded.

Discovering Entities Without Labels

A central challenge is learning these "slots" without human-provided bounding boxes or segmentation masks. The field has converged on Slot Attention and iterative competition as the primary mechanisms for unsupervised discovery.

In Object-Centric Multiple Object Tracking, Zixu Zhao and Max Horn demonstrate how iterative attention allows feature vectors to "compete" to explain parts of a scene. This competition forces the model to group pixels that move together or share appearance into a single slot. Similarly, Matthias Minderer and Kevin Patrick Murphy showed in Unsupervised learning of object structure and dynamics from videos that forcing a model through a spatial coordinate bottleneck—representing frames as a set of moving keypoints—naturally leads to the discovery of consistent moving entities.

Modeling Kinematics and Physics

Once a scene is decomposed into slots, the model can reason about how those slots change over time. Traditional models often predict the next frame's appearance directly, which leads to blurring. Object-centric models instead predict the evolution of the slots.

Suhyung Choi and colleagues introduced Object-Centric Kinematics (OCK) in Unsupervised Dynamics Prediction with Object-Centric Kinematics. By explicitly modeling attributes like velocity and acceleration within each slot, the model maintains temporal consistency even in complex scenes.

This representation also allows for the injection of "procedural knowledge." Patrick Takenaka and Marco F. Huber demonstrated in Guiding Video Prediction with Explicit Procedural Knowledge that a differentiable physics engine can act as a "shortcut" in the latent space. Because the model already sees the world as discrete objects, it can apply physical laws (like gravity or collisions) to specific slots, achieving stable predictions with 97% less training data.

From Prediction to Planning and Generation

The utility of object-centric representations extends beyond simple prediction into control and high-fidelity generation.

  • Controllable World Models: Angel Villar-Corrales and Sven Behnke developed PlaySlot (paper), which learns to infer "latent actions" from unlabeled video. By manipulating these actions at the slot level, they create a world model that a robot can use to plan complex interactions with objects.
  • Robust Video Generation: In the realm of text-to-video, Xingrui Wang and Xin Li proposed TIV-Diffusion (paper). By using Slot Attention to disentangle objects before generation, they prevent common artifacts where objects "melt" into each other or disappear during overlaps, as each entity's identity is maintained in its own dedicated data structure.

By moving away from the "pixel grid" dogma and toward a "collection of entities" framework, these researchers are building AI that understands the physical structure of the world, rather than just the statistical patterns of its light.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_object_centric_video_modeling_with_slot__36bd1115,
  title = {Object Centric Video Modeling with Slot Attention},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/object-centric-video-modeling-with-slot-attention/rdr_36bd1115a97b481a8577613379c1f5ac},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_object_centric_video_modeling_with_slot__36bd1115_v1,
  title = {Object Centric Video Modeling with Slot Attention},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/object-centric-video-modeling-with-slot-attention/rdr_36bd1115a97b481a8577613379c1f5ac/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}