Slot Order Matters for Compositional Scene Understanding

Patrick Emami, Pan He, Sanjay Ranka, Anand RangarajanarXiv 2022 · arXiv · OpenReview

Abstract

Unconditional scene inference and generation are challenging to learn jointly with a single compositional model. Despite encouraging progress on models that extract object-centric representations ("slots") from images, unconditional generation of scenes from slots has received less attention. This is primarily because learning the multi-object relations necessary to imagine coherent scenes is difficult. We hypothesize that most existing slot-based models have a limited ability to learn object correlations. We propose two improvements that strengthen slot correlation learning. The first is to condition the slots on a global, scene-level variable that captures higher-order correlations between slots. Second, we address the fundamental lack of a canonical order for objects by proposing to learn a consistent order to use for the autoregressive generation of scene objects. Specifically, we train an autoregressive slot prior to sequentially generate scene objects following the learned order. Slot inference entails estimating a randomly ordered set of slots using existing approaches for extracting slots from images, then aligning those slots to ordered slots generated autoregressively with the prior. Our experiments across three multi-object environments demonstrate clear gains in scene generation quality. Detailed ablation studies are also provided that validate the two proposed improvements.

Introduction

Object-centric representations, often called "slots," have become a cornerstone for compositional scene understanding. While models like MONet or Slot Attention excel at decomposing images into individual objects (the "binding problem"), the inverse task—unconditional scene generation—remains a significant challenge. Generating a coherent scene from scratch requires modeling complex multi-object correlations. For instance, in a stack of blocks, the position of the top block is strictly dependent on the blocks below it.

Existing slot-based Variational Autoencoders (VAEs) often treat slots as independent or fail to capture these higher-order dependencies. Furthermore, because sets of objects lack a natural canonical order, autoregressive models struggle to learn a consistent generation sequence, often leading to physically impossible scene configurations or "floating" objects.

The SRI Framework

To address these issues, we introduce the Segregate, Relate, Imagine (SRI) framework. SRI improves slot correlation learning through two primary mechanisms: a global scene-level variable and a learned canonical ordering for autoregressive generation.

The SRI framework consists of three stages: Segregate (extracting slots), Relate (computing a global scene variable), and Imagine (autoregressive generation in a learned order).

The framework operates in three distinct stages:

  1. Segregate: An encoder extracts a set of unordered slots from an input image. This stage handles the standard task of object decomposition.
  2. Relate: The model computes order-invariant correlations between these slots and summarizes them into a global scene-level latent variable ($s$). This variable acts as a "blueprint" for the entire scene, capturing higher-order correlations that individual slots might miss.
  3. Imagine: An autoregressive prior (typically an LSTM) generates slots sequentially. To train this, the authors use a greedy matching algorithm to align the randomly ordered slots from the "Segregate" step with the ordered slots from the "Imagine" step.

Crucially, the model is trained with a "cross-order" KL loss. This loss forces the autoregressive prior to learn a sequence that results in high-quality scenes, effectively teaching the model a "canonical" way to build a scene (e.g., "always draw the floor first, then the bottom block, then the top block").

Experiments and Results

We evaluated SRI on three synthetic multi-object benchmarks: Objects Room, CLEVR6, and ShapeStacks. ShapeStacks is particularly challenging as it requires modeling physical stability in block towers where contact points are critical.

Qualitative results showing the matching process between the segregation posterior and the imagined scene, alongside final reconstructions and masks.

The SRI framework, particularly the SRI-MoG variant (using Mixture-of-Gaussians), demonstrated substantial improvements over state-of-the-art baselines like GENESIS-v2:

  • Generation Quality: On ShapeStacks, SRI achieved a Fréchet Inception Distance (FID) of 70.4, a significant improvement over GENESIS-v2’s 112.7.
  • Structural Plausibility: The Structural Accuracy (S-Acc) on ShapeStacks reached 80.7%, compared to 59% for GENESIS-v2. This indicates the model successfully learned to generate physically stable towers.
  • Flexibility: SRI is "model-agnostic"—it can be wrapped around different slot-extraction backends (like EMORL) and still provide a significant boost in generation quality.

Discussion and Conclusion

The SRI framework demonstrates that for compositional scene understanding, slot order matters. By introducing a global scene variable and a learned canonical generation order, we bridge the gap between the permutation-invariant nature of object sets and the sequential nature of autoregressive priors.

While the model learns that objects are correlated, it does not yet explicitly model causal relationships (e.g., the physics of a falling tower). Future work may explore scaling this approach to more complex, real-world scenes and incorporating causal dynamics. Nonetheless, the ability to "imagine" coherent, structured scenes represents a major step forward for object-centric deep learning.

Summary updated

Abstract

Unconditional scene inference and generation are challenging to learn jointly with a single compositional model. Despite encouraging progress on models that extract object-centric representations ("slots") from images, unconditional generation of scenes from slots has received less attention. This is primarily because learning the multi-object relations necessary to imagine coherent scenes is difficult. We hypothesize that most existing slot-based models have a limited ability to learn object correlations. We propose two improvements that strengthen slot correlation learning. The first is to condition the slots on a global, scene-level variable that captures higher-order correlations between slots. Second, we address the fundamental lack of a canonical order for objects by proposing to learn a consistent order to use for the autoregressive generation of scene objects. Specifically, we train an autoregressive slot prior to sequentially generate scene objects following the learned order. Slot inference entails estimating a randomly ordered set of slots using existing approaches for extracting slots from images, then aligning those slots to ordered slots generated autoregressively with the prior. Our experiments across three multi-object environments demonstrate clear gains in scene generation quality. Detailed ablation studies are also provided that validate the two proposed improvements.
BibTeX
@misc{emami2022slot,
  author = {Patrick Emami and Pan He and Sanjay Ranka and Anand Rangarajan},
  title = {{Slot Order Matters for Compositional Scene Understanding}},
  year = {2022},
  eprint = {2206.01370},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2206.01370},
}
Figure