Learning Object Centric Representations from Raw Pixels

42 concepts39 papers

When a standard deep neural network processes a visual scene, it typically flattens the image into a grid of pixels or patches, eventually compressing the entire frame into a single, high-dimensional vector. This is known as a monolithic representation. While highly effective for simple image classification, this holistic approach creates a fundamental bottleneck for reasoning about dynamic environments.

Consider a robotic arm trying to arrange blocks on a table. In a monolithic world model, moving a single red block changes the entire latent vector. The network must learn the physics of that red block independently for every possible configuration of the other blocks. As highlighted by Yuan et al. in their survey on Compositional Scene Representation Learning via Reconstruction, this lack of structural decomposition prevents models from reasoning about individual objects, their specific attributes, or their interchangeable relationships. The model is forced to expend massive amounts of data to rediscover basic physical principles like object permanence and solidity—a limitation demonstrated empirically by Piloto et al. with the PLATO framework.

The Object-Centric Alternative: Slots and Particles

To overcome this, a growing body of research focuses on object-centric representation learning. Instead of encoding a scene into a single vector, these models decompose the visual input into a set of discrete latent variables, often referred to as "slots" or "particles." Each slot is responsible for capturing the properties (such as appearance, shape, and position) of exactly one object or background element.

How can a model learn to partition an image into these discrete slots without human-labeled masks or bounding boxes? The breakthrough lies in unsupervised competitive routing mechanisms.

In Object-Centric Learning with Slot Attention, Locatello et al. introduced an architectural module called Slot Attention. It uses an iterative attention mechanism where a set of randomly initialized vectors (the slots) compete to explain the features of an image. Because the attention weights are normalized across the slots via a softmax operation, a single pixel or patch can generally only be assigned to one slot. To minimize a reconstruction loss (e.g., re-generating the original image from the slots), the slots are forced to specialize, naturally binding to distinct, spatially coherent entities.

An earlier formulation of this piece-by-piece scene explanation was proposed by Burgess et al. in MONet, which pairs a recurrent attention network with a Variational Autoencoder (VAE) to progressively segment and encode objects. Similarly, Engelcke et al. developed GENESIS, which uses an autoregressive prior to capture the spatial relationships between these object-centric latents, preventing physical anomalies like two objects trying to occupy the same space.

Why It Works: Relational Symmetry and Compositional Generalization

The power of object-centric representations becomes obvious when modeling how a scene changes over time. Once an image is broken down into independent factors, we can apply the same neural network module to every slot symmetrically.

In Contrastive Learning of Structured World Models, Kipf et al. paired object-centric encodings with Graph Neural Networks (GNNs). In this setup, each slot acts as a node in a graph, and the GNN models the pairwise interactions between them. Because the GNN shares weights across all edges, the model learns a single, unified set of physical laws that applies to any object, regardless of which slot it occupies.

This architectural inductive bias enables compositional generalization. If a model learns how two blocks interact, it can immediately generalize that knowledge to a scene containing ten blocks, or to a completely novel spatial arrangement. As Biza et al. demonstrated in Factored World Models for Zero-Shot Generalization in Robotic Manipulation, this factored approach allows a robotic system to plan complex sequences of actions in environments it has never seen before, achieving true zero-shot generalization.

Enhancing Sample Efficiency in Reinforcement Learning

In visual Reinforcement Learning (RL), agents traditionally struggle because they must simultaneously learn to interpret raw pixels and discover optimal control policies. Object-centric models bridge this gap by providing a structured state space.

For instance, Co-Reyes et al. developed Entity Abstraction in Visual Model-Based Reinforcement Learning (OP3), enforcing the constraint that physical laws must apply symmetrically to all discovered entities. By planning in this object-abstracted space, the agent avoids the need to relearn dynamics when the number of objects changes.

More recently, Mosbach et al. introduced SOLD (Slot Object-centric Latent Dynamics), demonstrating that model-based RL agents operating on unsupervised slots significantly outperform monolithic state-of-the-art baselines on complex relational manipulation tasks. By isolating individual objects into slots, the agent can ignore task-irrelevant background noise or moving distractors—a benefit also leveraged by Chen et al. in their Focus-Then-Decide framework, which uses foundational segmentation models to filter out environmental noise before decision-making.

Alternative Granularities: Particles, Regions, and Categories

While abstract "slots" are a highly popular formulation, researchers have explored other granularities to capture objectness:

  • Latent Particles: Instead of abstract slots, Daniel et al. proposed Unsupervised Image Representation Learning with Deep Latent Particles (DLP), which represents scenes as a set of low-dimensional particles. Each particle explicitly disentangles an object's precise spatial coordinates from its local appearance. This was later extended to Latent Particle World Models by the authors to enable stochastic video prediction and imitation learning for robotics.
  • Region-Based Pretraining: Moving beyond synthetic or simple environments, Nguyen et al. introduced R-MAE, which injects an object-centric bias into self-supervised Masked Autoencoders by forcing the model to reconstruct masked region maps alongside raw pixels.
  • Object Categories: In Object-Category Aware Reinforcement Learning, Yi et al. argued that merely identifying discrete objects is not enough; agents must also understand object categories to effectively transfer functional knowledge (e.g., knowing that all objects categorized as "buttons" can be pressed).

By shifting from monolithic latent vectors to structured, object-centric representations, AI models gain a powerful inductive bias that mirrors human perception. Whether through slot-based attention, latent particles, or region-aware autoencoding, breaking the visual world into its constituent, interacting parts is proving essential for building data-efficient, generalizable, and physically intuitive world models.

Key Foundational Work

Object-Oriented Dynamics Predictor (2018) — contributed 1 concept elements to this direction.
Unsupervised learning of object structure and dynamics from videos (2019) — contributed 1 concept elements to this direction.
MONet: Unsupervised Scene Decomposition and Representation (2019) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_object_centric_representation_monolithic_63aca1b5,
  title = {object-centric representation (monolithic scene representation)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/object-centric-representation-monolithic-scene-representation/rdr_63aca1b530dc4dcdb1248ee92ceca3de},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_object_centric_representation_monolithic_63aca1b5_v1,
  title = {object-centric representation (monolithic scene representation)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/object-centric-representation-monolithic-scene-representation/rdr_63aca1b530dc4dcdb1248ee92ceca3de/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake