TL;DR
Object-Centric Learning (OCL) often struggles when objects appear at unfamiliar scales, as the Variational Autoencoders (VAEs) providing guidance have specific "comfort zones." This paper introduces Multi-Scale Fusion (MSF), which uses an image pyramid and a novel inter/intra-scale fusion mechanism to ensure objects of all sizes are represented with high confidence. This approach significantly boosts the performance of both transformer-based and diffusion-based OCL models.
The Challenge of Scale in Object-Centric Learning
In computer vision, Object-Centric Learning (OCL) aims to move away from processing raw pixels and toward representing scenes as a collection of "slots"—discrete feature vectors that each correspond to a single object. To train these models without manual labels, researchers often use a pre-trained VAE. The VAE provides an intermediate representation that acts as a blueprint, guiding the model to group pixels into meaningful objects.
However, there is a catch: VAEs are typically sensitive to scale. If an object is too small or too large relative to the patterns the VAE was trained on, the resulting guidance is noisy or low-quality. Because real-world objects vary wildly in size due to distance and perspective, a single-scale VAE often fails to provide the clear guidance needed to separate complex objects from their backgrounds.
Multi-Scale Fusion (MSF) Methodology
To bridge this scale gap, the authors propose Multi-Scale Fusion (MSF). The core idea is to ensure that objects of all sizes fall within the VAE's optimal processing range.
The approach consists of three main components:
- The Image Pyramid: Instead of feeding the model a single image, MSF creates a pyramid of three scales. By resizing the input, the system ensures that almost any object, regardless of its original size, will eventually fall into the VAE’s "comfort zone" at one of the pyramid levels.
- Inter-Scale Fusion (Scale-Invariance): The model projects VAE features into a scale-invariant space and looks across all levels of the pyramid for the same spatial region. By comparing "matching probabilities" (how confident the VAE is about a specific codebook entry), the system identifies which scale produced the highest quality representation for that patch. It then uses this high-confidence representation to augment the lower-quality scales.
- Intra-Scale Fusion (Scale-Variance): While some features are universal, others are specific to a certain size. The authors use separate, non-shared codebooks for these scale-specific details and fuse them back with the augmented scale-invariant features to create a final, high-fidelity representation.
Crucially, MSF uses an invertible "project-up-project-down" design, allowing the addition of these inductive biases without losing the original information density of the VAE.
Key Insights and Efficiency
A major insight of this work is using the VAE's quantization probability as a proxy for representation quality. If the VAE is highly confident about a patch at a smaller scale but unsure at the original scale, the model automatically trusts the smaller scale to fix the representation. This makes objects significantly more distinct and separable from the background before the OCL model even begins its slot aggregation.
Surprisingly, this method is also highly parameter-efficient. By splitting the codebook into smaller, specialized groups (invariant vs. variant), the authors use 81.25% fewer codebook parameters than standard baseline methods, despite processing multiple scales.
Experimental Results
The researchers evaluated MSF across a diverse set of benchmarks, including synthetic images (ClevrTex), real-world images (COCO and VOC), and synthetic videos (MOVi datasets).
- Performance Boosts: On the ClevrTex dataset, MSF improved the Adjusted Rand Index (ARI)—a measure of segmentation accuracy—for the SLATE model by roughly 12%.
- State-of-the-Art Compatibility: MSF is model-agnostic. It improved both transformer-based models (like SLATE and STEVE) and the current state-of-the-art diffusion-based model (SlotDiffusion), boosting its ARI on the VOC dataset by 3%.
- Visual Clarity: Qualitative results demonstrate that MSF-equipped models are vastly superior at identifying small objects and maintaining clean boundaries in complex, cluttered scenes where baseline models often bleed objects into the background.
Limitations and Future Work
While highly effective, MSF has limitations. On real-world datasets like COCO, performance gains were more modest because some objects (e.g., a massive field of grass) are so large they remain outside the coverage of even a three-scale pyramid. Additionally, processing three scales introduces roughly 31.25% more computation during the VAE pre-training phase.
Future work may explore integrating multi-scale fusion directly within the Slot Attention mechanism itself, rather than limiting it to VAE guidance, potentially leading to even more robust unsupervised object discovery.
Summary updated