Object-centric vision models split an image into a small set of "slots", which are vectors that each describe one object. The most widely used method, Slot Attention, builds these slots by repeating the same update a few times. On each pass, the slots compete to explain image regions and are then revised. Two findings from separate literatures meet at this loop. The first is that good object masks do not imply good object representations. Models can draw sharp segmentation boundaries while the slot vectors stay entangled and non-compositional. The second is that the refinement loop costs time and memory. Recent work therefore cuts it down, running one pass instead of three or starting the slots near their final value. Those shortcuts have been judged almost entirely by mask quality and reconstruction error. That is the one kind of metric the first literature says is unreliable.
We propose to test a specific hypothesis, which we call "masks settle before meaning". Within a single image's refinement, the attention masks stop changing after one or two passes. The slot vectors, meaning what each slot says about color, shape and position, may keep changing for longer, or may degrade. If so, any latency-saving rule that watches segmentation stops refinement too early for downstream use, and no current evaluation would notice. We will measure the two convergence rates separately. We will then build a per-image stopping rule driven by slot-content convergence rather than mask convergence, and ask whether it gives a better tradeoff between compute and compositional quality than fixed schedules, warm-started slots, or mask-based stopping.
Core Claim
The claim is that refinement latency and reconstruction-segmentation misalignment are one problem seen from two sides. The number of passes a model needs depends on what you measure convergence against. Existing acceleration work measures against the fast-settling quantity, the masks, and so may be saving compute at the expense of the slow-settling quantity, the object representation.
Concretely, at refinement step $t$ let $A_t$ be the slot-to-pixel attention masks and $S_t$ the slot vectors. We track two residuals:
$$ r^{\text{mask}}_t = \frac{\lVert A_t - A_{t-1}\rVert_1}{\lVert A_{t-1}\rVert_1}, \qquad r^{\text{slot}}_t = \frac{\lVert S_t - S_{t-1}\rVert_2}{\lVert S_{t-1}\rVert_2}. $$
Slot identity persists across iterations within one image, so the second residual needs no permutation matching.
The prediction has three parts. The first is that $r^{\text{mask}}$ falls below a small threshold one or more steps before $r^{\text{slot}}$ does. The second is that the gap is larger on texture-rich scenes, where reconstruction loss rewards fragmentation. The third is that compositional and disentanglement scores keep moving over steps in which mask agreement with ground truth (foreground Adjusted Rand Index, FG-ARI) is flat.
If the hypothesis holds, the new artifact is a halting rule on $r^{\text{slot}}$, together with a warm-start module distilled toward content-converged slots. If the hypothesis fails, and slot content settles no later than masks, that is also a useful result. It would mean the one-pass shortcuts are safe for compositional use, which nobody has shown.
Research Program
Motivating evidence. The idea rests on three results about misalignment.
Evaluating Disentanglement of Structured Representations introduced object-level and property-level disentanglement probes on Multi-dSprites and CLEVR6. It found a negative correlation between ARI and latent disentanglement.
COAT: Measuring Object Compositionality in Emergent Representations tests whether slot vectors behave like algebra under object swaps, meaning whether $z_B - z_A \approx z_D - z_C$. It reports that raw Slot Attention scores 48.55% on $L_2$-COAT, below a raw-pixel baseline at 75.47%, and rises to 77.02% only after slot post-processing.
Evaluating Object-Centric Models beyond Object Discovery shows that models that are strong at unsupervised discovery can still be weak at downstream reasoning.
None of these papers looks inside the refinement loop. They evaluate only the final slots.
Methods we reuse, and the gap they leave.
Object-Centric Learning with Slot Attention supplies the loop itself. It uses slot-normalized attention followed by a recurrent update, usually run for $T=3$ passes.
Object Representations as Fixed Points treats the loop's endpoint as a fixed point. It trains the loop with implicit differentiation, a single backward step at the fixed point that replaces backpropagation through every pass, and so lets the forward pass run to convergence at constant memory. Its evaluation is reconstruction error and segmentation.
Smoothing Slot Attention Iterations and Recurrences (SmoothSA) adds a small "preheater" network. The preheater is trained by self-distillation to predict refined slots from the image, so that fewer passes are needed. SmoothSA uses a fixed 3-then-1 schedule in video and is evaluated on segmentation and question answering.
Both acceleration papers frame the loop as a contraction toward one fixed point. Neither asks whether the masks and the content reach that point at the same rate. Our stopping rule and our content-targeted distillation are the parts they lack.
Constraints from prior work. Top-Down Guidance for Learning Object-Centric Representations adds an inference-time loop that splits and merges slots using distance thresholds. That loop raises latency, and it warns us against adding open-ended test-time loops. GENESIS-V2 drops iteration entirely by clustering in a single pass. It oversegments complex scenes and reaches an ARI of only 0.55 on the Amazon Picking Challenge data. This caps how far "zero iterations" can be pushed.
What we build. The program has three components.
- Instrumentation. We add hooks to Slot Attention and SLATE-style models at every pass. The loops are run to a long horizon ($T$ up to 10) at inference, and trained both unrolled and with implicit differentiation. At each pass we record FG-ARI, the two residuals, COAT scores, and the property-level disentanglement probes.
- Halting rule. We add a per-image rule that stops when $r^{\text{slot}}_t$ drops below a threshold $\epsilon$. The threshold is swept to trace a curve of compute against quality. We also test a per-slot variant, in which settled slots are frozen while unsettled ones continue.
- Content-targeted preheater. This is a SmoothSA-style warm-start module distilled toward slots taken at the content-converged step, not a fixed $T$. The aim is to test whether the warm start can carry the late-settling content, rather than only the early-settling masks.
Datasets and models. We use four datasets. Multi-dSprites and CLEVR6 give access to the disentanglement probes. CLEVRTex, from ClevrTex, is where texture-driven fragmentation is worst. The COAT evaluation set is used for compositional algebra. Our backbones are pixel-reconstructing Slot Attention and a DINO-feature-reconstructing variant. We include the second because feature targets may change the mask-content gap.
Baselines
Do nothing: standard Slot Attention. The naive reference is standard Slot Attention at its default $T=3$, trained by unrolled backpropagation, with no halting rule. It fixes the cost floor that every other method is compared against. The COAT paper reports raw Slot Attention at 48.55% $L_2$-COAT, and we will first reproduce that figure in our setting.
Strongest latency competitor: SmoothSA. The strongest latency competitor is SmoothSA, in two forms: with its preheater and one pass, and with its preheater and three passes. It is the only method in the file that cuts passes and reports improved segmentation. It is exactly what our hypothesis says may be trading away content.
We rerun it from the authors' description on our datasets, with the same encoder, the same slot count, the same training steps, and an equal hyperparameter-search budget. We found no reported result on Lacuna for SmoothSA on COAT, CLEVRTex or disentanglement probes. Its reported gain is more than 4 accuracy points on GQA over prior object-centric models.
Convergence-to-fixed-point reference: Implicit Slot Attention. Implicit Slot Attention, run to convergence at inference, is the second key comparison. It represents "just iterate until done" without the unrolling instability. Object Representations as Fixed Points reports that Implicit SLATE reaches a reconstruction error of 10.66 on CLEVR-Mirror, versus 67.04 for unrolled SLATE.
This baseline is the upper bound on content quality at high cost. Our halting rule beats it if it reaches that baseline's COAT and disentanglement scores, within its run-to-run spread, using meaningfully fewer passes on average.
Key ablation: mask-residual halting. The ablation that isolates our ingredient is an otherwise identical halting rule triggered by $r^{\text{mask}}$ instead of $r^{\text{slot}}$. It is swept over thresholds so that it spends the same average number of passes. If the two rules produce the same compute-quality curve, the content signal adds nothing and the core claim fails.
Post-processing reference. Finally, we apply COAT's reported slot post-processing on top of each method. This separates gains from better refinement from gains that any post-hoc fix would provide.
Winning criterion. We count the proposal as winning if the slot-residual rule dominates both SmoothSA and mask-residual halting on at least two of three content metrics at matched average compute. The metrics are COAT score, property-level disentanglement, and attribute-probe accuracy. It must do so without losing more than one FG-ARI point.
Why not just…?
Why not just keep the fixed three-pass schedule? Fixed $T$ is simple, and it may already be enough. The instrumentation experiment settles this directly. If content residuals are near zero by pass three on every dataset, there is no room for a smarter rule except saving compute on easy images. We would report that as a limited positive result for current practice. The raw COAT score of 48.55%, falling below pixels, hints that three passes do not produce compositional slots. But it does not show that more passes would help, which is why we measure the trajectory.
Why not just post-process the slots, as COAT does? This is the most serious threat. COAT reports that post-processing lifts Slot Attention from 48.55% to 77.02%, which is more than any iteration change is likely to deliver. If post-processing erases every difference between refinement schedules, then the schedule does not matter for compositionality, and our proposal reduces to a latency trick. We therefore report every result both before and after post-processing. The claim survives only if content-aware halting still helps after post-processing, or if it reaches the post-processed score with less work.
Why not just halt when the masks stop changing? This is the obvious cheap rule, and it is exactly our key ablation. The hypothesis predicts it stops too early on CLEVRTex, where Inductive Biases for Object-Centric Representations in the Presence of Complex Textures shows reconstruction loss pushes toward texture-driven splitting and merging. If it matches our rule, we drop the claim.
Why not just iterate to convergence with implicit differentiation? This may deliver the best content, but it pays full latency on every image. Its convergence also rests on an assumed contraction that has not been checked on textured scenes. Our experiments measure whether slot content actually converges or oscillates under it. They also measure how many passes can be saved per image without losing the endpoint's quality.
Decisive Experiments
Experiment 1: the per-pass trajectory study. For each model, dataset and training method, we plot FG-ARI, $L_2$-COAT, and property-level disentanglement against pass index up to $T=10$. Beside these we plot the two residuals. The hypothesis is confirmed if, on CLEVRTex and CLEVR6, COAT or disentanglement changes by a margin beyond seed variance, across passes where FG-ARI has plateaued. It is refuted if all curves flatten together.
Experiment 2: compute-quality curves. We compare compute-quality curves across five rules: fixed $T$, SmoothSA, mask-residual halting, slot-residual halting, and the content-targeted preheater. Compute is measured as mean passes and as wall-clock time per image on the same hardware. The winning criterion is the one stated under Baselines.
Experiment 3: harder regimes. We repeat the curves on DINO-feature-reconstructing models. We also stratify results by object count and texture difficulty. This tests whether the mask-content gap tracks the texture failure modes documented in ClevrTex, where SPAIR's mIoU falls from about 66% on CLEVR to 0%.
A positive outcome would give object-centric models a stopping criterion that matches what downstream reasoning needs, rather than what segmentation benchmarks reward. A negative outcome would, for the first time, certify that the existing one-pass shortcuts cost nothing in compositionality.
References
- Evaluating Disentanglement of Structured Representations
- COAT: Measuring Object Compositionality in Emergent Representations
- Evaluating Object-Centric Models beyond Object Discovery
- Object-Centric Learning with Slot Attention
- Object Representations as Fixed Points: Training Iterative Refinement Algorithms with Implicit Differentiation
- Smoothing Slot Attention Iterations and Recurrences
- Top-Down Guidance for Learning Object-Centric Representations
- GENESIS-V2: Inferring Unordered Object Representations without Iterative Refinement
- ClevrTex: A Texture-Rich Benchmark for Unsupervised Multi-Object Segmentation
- Inductive Biases for Object-Centric Representations in the Presence of Complex Textures