Smoothing Slot Attention Iterations and Recurrences

Rongzhen Zhao, Wenyan Yang, Juho Kannala, Joni Pajarinen

Introduction

Object-Centric Learning (OCL) aims to decompose complex visual scenes into discrete "slots," each representing an individual object. The primary mechanism for this task is Slot Attention (SA), an iterative cross-attention process where query vectors compete to extract information from image features. Despite its success, standard SA implementations suffer from two significant inefficiencies: the "cold-start" problem and transform homogeneity.

The Cold-Start Issue

In the initial frame of a video or a static image, queries are typically sampled from a global Gaussian distribution. These "cold" queries lack any scene-specific context, requiring multiple iterations of the attention mechanism simply to locate objects within the feature space.

Transform Homogeneity

While subsequent video frames benefit from "warm" queries transitioned from previous slots, standard models apply the same number of iterations and identical transformation parameters across all frames. This approach is computationally expensive and fails to leverage the high-quality information already present in temporal recurrences.

Methods: The SmoothSA Framework

To address these bottlenecks, the authors introduce SmoothSA, featuring two primary architectural innovations: Query Preheating and Differentiated Recurrences.

The SmoothSA architecture featuring the preheater module and differentiated iteration counts for first vs. non-first frames.

Query Preheating

SmoothSA incorporates a "preheater" module—a lightweight Transformer decoder—that processes initial cold queries alongside image features. This module produces "preheated" queries that are already conditioned on the specific sample. The preheater is trained via self-distillation, learning to predict the final slot outputs of the main SA module. This bootstrapping process allows the model to reach high-quality object representations much faster without requiring external labels.

Differentiated Recurrences

SmoothSA optimizes video processing by varying the computational effort based on query quality. The model utilizes three iterations of SA for the first frame to refine preheated queries but reduces this to a single iteration for all subsequent frames. Because transitioned queries are already highly informative, a single update is sufficient, significantly reducing computational overhead without sacrificing accuracy.

Experimental Evaluation

The model was evaluated on diverse benchmarks, including synthetic (ClevrTex) and real-world (COCO, VOC) images, as well as video datasets (YTVIS, MOVi-C). Downstream reasoning was tested using Visual Question Answering (VQA) tasks on GQA and CLEVRER.

Key Results and Insights

SmoothSA achieves new state-of-the-art performance in object discovery and downstream reasoning:

  • Superior Discovery: On the YTVIS video dataset, SmoothSA improved the Adjusted Rand Index (ARI) from 42.0 to 44.1. Similar gains were observed on the VOC dataset, with a 4-point ARI increase over previous methods.
  • Enhanced Downstream Reasoning: Integrating SmoothSA slots into the Aloe VQA model led to a 4.4% accuracy boost on the GQA dataset, indicating that the learned slots capture more semantic detail.
  • Efficiency Gains: By reducing iterations on non-first frames, the model cuts the computational cost of the aggregation stage by approximately 2/3 for the majority of video sequences.
Qualitative comparison of object discovery performance on images and video sequences, showing SmoothSA's ability to produce cleaner and more accurate masks.

Conclusion and Future Work

SmoothSA demonstrates that sample-specific initialization and adaptive processing are key to efficient and effective Object-Centric Learning. While the model significantly improves performance, it still relies on a fixed number of slots. Future research will explore dynamic slot allocation to better handle scenes with varying object counts.

Summary updated

BibTeX
@inproceedings{zhao2026smoothing,
  author = {Rongzhen Zhao and Wenyan Yang and Juho Kannala and Joni Pajarinen},
  title = {{Smoothing Slot Attention Iterations and Recurrences}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=TXV35hR2Ib},
}
Figure