Predicting Video Slot Attention Queries from Random Slot-Feature Pairs

Rongzhen Zhao, Jian Li, Juho Kannala, Joni Pajarinen

Abstract

Unsupervised video object-centric learning aims to decompose video frames into individual object representations, or "slots." Current state-of-the-art models often struggle with temporal consistency due to poorly performing transitioner modules. This paper introduces RandSF.Q, a novel approach that improves query prediction by incorporating future frame features and employing a random time-interval training strategy. RandSF.Q achieves significant improvements in object discovery, notably a 10-point gain on the YTVIS dataset.

Introduction: The "Eyes-Closed" Transitioner Problem

In Object-Centric Learning (OCL), models typically use a recurrent loop where an aggregator identifies objects in the current frame ($t$) and a transitioner predicts their state in the next frame ($t+1$) to guide the next aggregation step. However, the authors identify two critical flaws in this paradigm.

First, existing transitioners suffer from "Information Blindness." They attempt to predict the next query using only the current slots, ignoring the features of the next frame which are already available during the forward pass. This is akin to trying to guess the future with one's eyes closed. Second, these models often fail to learn true scene dynamics because they are trained only on consecutive frames. Empirically, the authors observed that for some models like VideoSAUR, removing the transitioner entirely actually improved performance, indicating that the module was providing detrimental noise rather than useful predictions.

Analysis of transitioner limitations and the proposed informative query prediction.

Methods: RandSF.Q

To address these issues, the authors propose RandSF.Q (Random Slot-Feature pair for learning Query prediction). The method introduces two primary innovations:

1. Informative Query Prediction

Instead of a blind prediction, the RandSF.Q transitioner takes both the current slots ($S_t$) and the next frame's features ($F_{t+1}$) as input. By conditioning the query on the target frame's features, the model performs an "informed search" rather than a "blind guess," leading to much more accurate object localization in the next step.

2. Effective Query Prediction Learning

To force the model to learn robust dynamics, RandSF.Q moves beyond consecutive frame training. During training, the model randomly samples pairs of slots and features from a wider time window $\Delta$. For example, it might use slots from frame $t-3$ to predict the query for frame $t$. This random sampling acts as a strong inductive bias, requiring the transitioner to understand how objects evolve over varying time scales. To support this, relative time embeddings are injected into the features to inform the model of the temporal distance between the slots and the target features.

The RandSF.Q architecture and random time-interval sampling mechanism.

The architecture utilizes a frozen DINOv2 backbone for feature extraction and a Transformer-based transitioner, building upon the SlotContrast framework.

Key Results

RandSF.Q was evaluated on synthetic (MOVi-C/D, CLEVRER) and real-world (YTVIS) datasets, setting new state-of-the-art benchmarks:

  • Object Discovery: On the challenging YTVIS dataset, RandSF.Q outperformed the previous state-of-the-art (SlotContrast) by over 10 points in Adjusted Rand Index (ARI) and ARI-foreground (ARIfg).
  • Downstream Performance: The improved object representations translated to better performance in object recognition (Top-1 accuracy on YTVIS rose from 19.9% to 26.1%) and Visual Question Answering on CLEVRER, demonstrating that better tracking leads to superior scene understanding.

Conclusion and Future Work

RandSF.Q demonstrates that query prediction in video slot attention is significantly enhanced by providing the transitioner with target frame context and training across random time intervals. While the model currently relies on a fixed number of slots—a limitation when objects enter or leave a scene—it provides a powerful foundation for future work in dynamic, adaptive object-centric video analysis.

Summary updated

BibTeX
@inproceedings{zhao2026predicting,
  author = {Rongzhen Zhao and Jian Li and Juho Kannala and Joni Pajarinen},
  title = {{Predicting Video Slot Attention Queries from Random Slot-Feature Pairs}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2026},
  doi = {10.1609/aaai.v40i16.38322},
}
Figure