Abstract
Unsupervised video object-centric learning aims to decompose video frames into individual object representations, or "slots." Current state-of-the-art models often struggle with temporal consistency due to poorly performing transitioner modules. This paper introduces RandSF.Q, a novel approach that improves query prediction by incorporating future frame features and employing a random time-interval training strategy. RandSF.Q achieves significant improvements in object discovery, notably a 10-point gain on the YTVIS dataset.
Introduction: The "Eyes-Closed" Transitioner Problem
In Object-Centric Learning (OCL), models typically use a recurrent loop where an aggregator identifies objects in the current frame ($t$) and a transitioner predicts their state in the next frame ($t+1$) to guide the next aggregation step. However, the authors identify two critical flaws in this paradigm.
First, existing transitioners suffer from "Information Blindness." They attempt to predict the next query using only the current slots, ignoring the features of the next frame which are already available during the forward pass. This is akin to trying to guess the future with one's eyes closed. Second, these models often fail to learn true scene dynamics because they are trained only on consecutive frames. Empirically, the authors observed that for some models like VideoSAUR, removing the transitioner entirely actually improved performance, indicating that the module was providing detrimental noise rather than useful predictions.
Methods: RandSF.Q
To address these issues, the authors propose RandSF.Q (Random Slot-Feature pair for learning Query prediction). The method introduces two primary innovations:
1. Informative Query Prediction
Instead of a blind prediction, the RandSF.Q transitioner takes both the current slots ($S_t$) and the next frame's features ($F_{t+1}$) as input. By conditioning the query on the target frame's features, the model performs an "informed search" rather than a "blind guess," leading to much more accurate object localization in the next step.
2. Effective Query Prediction Learning
To force the model to learn robust dynamics, RandSF.Q moves beyond consecutive frame training. During training, the model randomly samples pairs of slots and features from a wider time window $\Delta$. For example, it might use slots from frame $t-3$ to predict the query for frame $t$. This random sampling acts as a strong inductive bias, requiring the transitioner to understand how objects evolve over varying time scales. To support this, relative time embeddings are injected into the features to inform the model of the temporal distance between the slots and the target features.
The architecture utilizes a frozen DINOv2 backbone for feature extraction and a Transformer-based transitioner, building upon the SlotContrast framework.
Key Results
RandSF.Q was evaluated on synthetic (MOVi-C/D, CLEVRER) and real-world (YTVIS) datasets, setting new state-of-the-art benchmarks:
- Object Discovery: On the challenging YTVIS dataset, RandSF.Q outperformed the previous state-of-the-art (SlotContrast) by over 10 points in Adjusted Rand Index (ARI) and ARI-foreground (ARIfg).
- Downstream Performance: The improved object representations translated to better performance in object recognition (Top-1 accuracy on YTVIS rose from 19.9% to 26.1%) and Visual Question Answering on CLEVRER, demonstrating that better tracking leads to superior scene understanding.
Conclusion and Future Work
RandSF.Q demonstrates that query prediction in video slot attention is significantly enhanced by providing the transitioner with target frame context and training across random time intervals. While the model currently relies on a fixed number of slots—a limitation when objects enter or leave a scene—it provides a powerful foundation for future work in dynamic, adaptive object-centric video analysis.
Summary updated