TL;DR
Hierarchical Slot Attention (HSA) bridges the gap between spatial appearance and human semantic hierarchies in object-centric learning. By combining a shared DINOv2 backbone with three level-specific slot attention modules and minimal supervision (10% labels), HSA decomposes scenes simultaneously at holistic, semantic, and panoptic levels within a single forward pass, significantly outperforming flat baselines in both accuracy and efficiency.
What Problem Does This Paper Address?
Object-centric learning (OCL) aims to break down visual scenes into discrete entity representations, typically called "slots." While recent slot attention methods excel at grouping pixels based on visual appearance (often leveraging self-supervised features like DINOv2), they suffer from two major flaws: they are "flat" (operating at a single granularity) and purely appearance-driven.
Humans, however, perceive scenes hierarchically and semantically. We naturally separate foreground from background (holistic level), group objects by category (semantic level), and distinguish individual instances within those categories (panoptic level). Because semantic categories are human constructs rather than raw visual patterns, true semantic hierarchies cannot emerge purely from unsupervised appearance reconstruction. HSA addresses this limitation by introducing a minimal grounding signal to guide multi-granularity scene decomposition.
Methods
The authors propose Hierarchical Slot Attention (HSA), a framework that replaces the traditional single flat slot aggregator with three parallel, level-specific slot attention modules operating on top of a shared, frozen DINOv2 ViT backbone.
The model is jointly optimized using three distinct objectives:
- Feature Reconstruction ($\mathcal{L}_{\text{recon}}$): Each level utilizes an independent Transformer decoder to reconstruct the self-supervised DINOv2 patch features. This regularizes the slots to retain complete scene information and prevents representation collapse.
- Segmentation Supervision ($\mathcal{L}_{\text{sup}}$): To ground the slots semantically, a Hungarian-matched Dice loss is applied to a small fraction (10%) of labeled data across all three granularity levels.
- Hierarchical Alignment ($\mathcal{L}_{\text{align}}$): To ensure structural coherence, this novel loss enforces cross-level embedding consistency between consecutive layers (holistic-to-semantic and semantic-to-panoptic). It computes a pairwise cosine similarity and uses a straight-through estimator (STE) to differentiably map fine-grained slots to their nearest coarse-grained counterparts.
Datasets
The framework was evaluated on two standard computer vision benchmarks:
- MS COCO 2017: A highly complex dataset featuring diverse real-world images with multiple co-occurring objects at varying scales, making it a rigorous testbed for simultaneous multi-level understanding.
- PASCAL VOC 2012: Comprises images with fewer, more salient objects, providing a complementary evaluation setting.
Key Insights and Ideas
- Supervision as a Catalyst: Pure appearance-based reconstruction is under-constrained. However, adding a minimal symbolic grounding signal (even just 1% labeled data) acts as a powerful catalyst to bootstrap semantically meaningful hierarchies.
- The Failure of Flat Supervision: Simply providing 10% supervision to traditional "flat" slot attention models often degrades their performance. Without explicit cross-level context and alignment losses, independent supervision conflicts with the spatial reconstruction objective. Joint hierarchical training is essential.
- New Structural Metrics: To prove that HSA learns a genuine semantic hierarchy in its representation space (and not just matching masks), the authors introduce two new metrics: attention containment (measuring spatial nesting) and grouping purity (measuring if fine-level slots assigned to a coarse slot share the same ground-truth category).
Key Results
- Significant Performance Gains: On the COCO dataset, HSA outperforms the strongest flat baseline (SPOT) by +41.5 ARI at the holistic level, +14.6 ARI at the semantic level, and +10.4 ARI at the panoptic level.
- High Efficiency: Instead of training and running three separate flat models for different granularities, HSA achieves full three-level decomposition in a single forward pass. It reduces inference time to 17.3 ms (compared to 30–88 ms for baselines) and significantly lowers parameter overhead.
- Qualitative Grounding: Qualitative visualizations demonstrate that HSA cleanly separates complex scenes into human-aligned categories and instances, whereas unsupervised baselines produce semantically arbitrary fragments.
Limitations and Future Work
- Metric Penalization: Standard pixel-clustering metrics like the Adjusted Rand Index (ARI) strictly penalize the model when it discovers valid, fine-grained object boundaries that are absent from the ground-truth annotations (such as separating a plate from food, or ski poles from a skier).
- Future Directions: The authors plan to explore even weaker grounding signals, such as natural language descriptions instead of pixel-level segmentation masks, to further reduce the supervision requirement. They also aim to extend HSA to video data for temporal hierarchical decomposition.
Summary updated