Understanding Clustering in Slot Attention via Particle Dynamics

Vasudev Joy, Rajat Rasal, Avinash Kori, Anthea Monod, Ben GlockerarXiv 2026 · arXiv

TL;DR

This paper demonstrates that the impressive object-centric clustering seen in Slot Attention is largely driven by its underlying competitive dynamics rather than its learned neural network components. By introducing a parameter-free "Simplified Slot Attention" (SSA) that behaves like a dot-product version of soft $k$-means, the authors achieve segmentation performance on par with the full model, suggesting that the "intelligence" of the system lies in the physics of the interaction.

What Problem Does This Paper Address?

Object-centric learning aims to decompose complex scenes into individual entities or "slots." Slot Attention is a popular architectural component for this task, but it is a "black box" containing several learned parts: query/key/value projections, Gated Recurrent Units (GRUs), and MLPs.

The authors ask a fundamental question: How much of the clustering behavior is actually learned, and how much is intrinsic to the attention mechanism itself? By viewing attention as an interacting particle system—where "slots" are particles competing to explain "input features"—they aim to provide a mechanistic explanation for why these models successfully segment objects.

Methods

The authors derive Simplified Slot Attention (SSA) by stripping away all learned parameters from the original Slot Attention module:

  1. Identity Projections: They set the Query ($Q$), Key ($K$), and Value ($V$) matrices to the identity matrix ($I$), meaning slots and inputs interact directly via their raw features.
  2. No Recurrence or MLPs: They replace the GRU and residual MLP updates with a simple weighted-mean aggregation.
  3. Competitive Dynamics: SSA retains the "column-stochastic" normalization of slot attention, where slots compete for each input feature.

The resulting update rule for a slot $s_i$ at time $t+1$ is: $$s_i^{t+1} = \frac{\sum_j w_{ij}^t x_j}{\sum_j w_{ij}^t}$$ where the weights $w_{ij}$ are determined by the softmax of the dot product between slots and inputs.

The authors also analyze USSA (Unnormalised SSA), a surrogate model that uses a weighted sum instead of a mean, to derive an energy function that characterizes the system's fixed points.

Datasets

The models were evaluated on the Pascal VOC 2012 dataset. To isolate the grouping mechanism, the authors used a frozen DINO ViT-B/16 encoder to provide input patch features. This ensures that any differences in performance come from the clustering module itself, not from a backbone being co-trained to "help" a specific clustering method.

Key Insights and Ideas

  • Dot-Product vs. Euclidean Distance: SSA is mathematically similar to Soft $k$-Means (SKM), but it uses dot-product similarity instead of Euclidean distance. The authors show that dot-product similarity allows the model to split clusters based on the angle between features, which is particularly effective for high-dimensional self-supervised features like DINO.
  • The Role of the Weighted Mean: While USSA (weighted sum) tends to collapse, causing slots to "die out" until only one or two remain (see Figure 4), the weighted mean in SSA prevents this collapse and maintains a diverse set of active slots.
  • Dynamics > Learning: The fact that a parameter-free model (SSA) performs nearly as well as a fully learned one suggests that the learned components in Slot Attention are primarily refining representations rather than constructing the clustering mechanism from scratch.
Figure 4: Evolution of segmentations under USSA. Note how slots progressively "die out" until only the cat and background remain.

Key Results

In quantitative tests (Table 1), SSA proved to be remarkably competitive:

  • mBO (mean Best Overlap): SSA achieved an instance-level mBO ($mBO_i$) of 37.369, very close to the full Slot Attention's 38.257.
  • Semantic Segmentation: On class-level overlap ($mBO_c$), SSA actually slightly outperformed the full Slot Attention (40.920 vs 40.790).
  • Reconstruction: SSA's reconstruction MSE (0.390) was only slightly higher than the full model (0.382), significantly beating standard $k$-means and other variants.

Qualitatively, Figure 1 shows that SSA produces segmentation masks that are visually indistinguishable from the much more complex Slot Attention model.

Figure 1: Pascal VOC segmentations across methods. Top: cluster assignments; Bottom: decoder alpha masks. Colours denote predicted slots and are not matched across methods. We provide additional qualitative examples in Section A.1.

Limitations and Future Work

  • Theoretical Gap: The energy function analysis was performed on USSA (the unnormalized version), leaving a slight gap in the formal mathematical characterization of SSA's specific weighted-mean dynamics.
  • Scope of Tasks: The study was limited to a single dataset (Pascal VOC) and a reconstruction-based training setup. It remains to be seen if parameter-free clustering holds up in more demanding generative tasks or complex video datasets.
  • Heuristic Explanations: The roles of the omitted components (like the GRU damping) are discussed heuristically. Future work could systematically reintroduce these parts to quantify exactly how much "damping" or "similarity learning" contributes to convergence speed and stability.

Summary updated

Abstract

Studying attention through the lens of interacting particle dynamics has shown how token clustering can emerge from the underlying dynamics. We extend this perspective to slot attention, a method for object-centric image segmentation and representation learning in which learned components obscure how much of the clustering behaviour is intrinsic to the attention dynamics. We therefore introduce simplified slot attention (SSA), a parameter-free variant whose dynamics are connected to soft $k$-means clustering and which provides a straightforward mechanistic explanation for the emergence of object-centric representations. On the Pascal VOC dataset, SSA achieves performance comparable to that of slot attention, demonstrating that competitive object-centric segmentation can be achieved without learned neural-network components.
BibTeX
@misc{joy2026understanding,
  author = {Vasudev Joy and Rajat Rasal and Avinash Kori and Anthea Monod and Ben Glocker},
  title = {{Understanding Clustering in Slot Attention via Particle Dynamics}},
  year = {2026},
  eprint = {2610.04493},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.04493},
}
Figure