Learning Object Centric Representations with Slot Attention

362 concepts237 papers1 proposal

Standard neural networks typically represent images as monolithic grids of pixels or high-dimensional feature maps. While effective for classification, these "distributed" representations struggle with the basic human intuition that a scene is composed of individual, interacting entities. Object-centric learning aims to bridge this gap by decomposing visual scenes into a set of discrete, exchangeable latent variables, often called slots.

The Bottleneck of Competition

The core challenge in unsupervised scene decomposition is forcing a model to group related pixels—like those belonging to a single coffee mug—into one representation while keeping them separate from the table beneath it. Locatello et al. introduced Slot Attention to solve this through a competitive attention mechanism.

Unlike standard attention, where multiple queries can attend to the same key, Slot Attention applies a softmax normalization across the slots (the queries). This creates a competitive bottleneck: each part of the input feature map must "choose" which slot it belongs to. As Locatello argues, this iterative process allows a set of initially random latent vectors to specialize, with each slot eventually binding to a distinct object in the scene.

Why It Works: Inductive Biases for Objects

The success of slot-based models relies on specific architectural choices that encourage decomposition:

  • Permutation Invariance: Slots are treated as a set rather than a sequence. This allows the model to handle a variable number of objects and generalize to new scene compositions that it did not see during training, a property highlighted by Huang et al..
  • Spatial Broadcasting: To reconstruct an image from slots, models often use a "spatial broadcast" decoder. Each slot is tiled into a grid and augmented with coordinate information. This forces each slot to capture localized, object-specific information to successfully reconstruct its assigned portion of the image, as described by Locatello et al..
  • Component Isolation: By isolating entities into slots, models prevent "interference" between unrelated components. Dedhia et al. found that this decomposition is essential for compositional tasks, such as changing the color of one object without affecting others.

Scaling to the Real World

Early object-centric models were largely confined to simple synthetic datasets with plain backgrounds. Recent work has focused on bridging this gap to complex, real-world environments.

Seitzer et al. proposed DINOSAUR, which shifts the reconstruction target from raw pixels to high-level semantic features from self-supervised models like DINO. By reconstructing features rather than pixels, the model avoids getting bogged down in low-level textures and instead focuses on meaningful semantic entities. Similarly, Elsayed et al. demonstrated with SAVi++ that incorporating depth signals (like LiDAR) and temporal consistency allows these models to scale to the messy reality of autonomous driving videos.

Beyond Static Images

Object-centric representations are particularly powerful for modeling dynamics and reasoning. When a scene is represented as a collection of slots, predicting the future becomes a matter of modeling how individual slots change over time.

  • Video Prediction: Hsieh showed that decomposing a video into independent components allows for predicting low-dimensional trajectories rather than high-dimensional pixels, making it easier to model complex interactions like collisions.
  • Action Recognition: By distilling video frames into compact object tokens, Qian et al. achieved state-of-the-art action recognition with significantly fewer parameters, as the model could focus on the movement of specific entities rather than the entire background.
  • Robotics: In manipulation tasks, Ye and Giannakakis et al. demonstrated that object-centric encoders allow robots to plan multi-step tasks more efficiently by reasoning about the spatial relationships between discrete objects.

By treating the world as a collection of entities rather than a grid of data, these models gain a structural inductive bias that mirrors human perception, leading to better generalization, transparency, and efficiency.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 271 papers
  2. Version 1 · 2026-10-04 · 386 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_object_centric_representation_object_cen_7ed2e313,
  title = {object-centric representation (object-centric latent representation)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/object-centric-representation-object-centric-latent-representation/rdr_7ed2e31315f24ee691e4c2e67010c871},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 2, frozen text):

@misc{lacuna_object_centric_representation_object_cen_7ed2e313_v2,
  title = {object-centric representation (object-centric latent representation)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/object-centric-representation-object-centric-latent-representation/rdr_7ed2e31315f24ee691e4c2e67010c871/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake