Multi-Scale Fusion for Object Representation

Rongzhen Zhao, Vivienne Wang, Juho Kannala, Joni Pajarinen

Multi-Scale Fusion for Object Representation

Abstract

Object-Centric Learning (OCL) aims to decompose complex scenes into discrete, manageable "slots." However, existing models often struggle with objects of varying sizes due to the scale-sensitivity of their internal guidance systems, typically Variational Autoencoders (VAEs). This paper introduces Multi-Scale Fusion (MSF), a novel framework that leverages an image pyramid and a specialized fusion mechanism to ensure high-quality object representations across all scales. By identifying and augmenting features within their respective "comfort zones," MSF significantly enhances object discovery performance on both synthetic and challenging real-world benchmarks like COCO and VOC.

Introduction: The Scale Challenge in Object-Centric Learning

The goal of Object-Centric Learning (OCL) is to move beyond simple pixel-level segmentation toward a more structured representation of scenes as collections of discrete entities or "slots." To achieve this without human supervision, many state-of-the-art models rely on a Variational Autoencoder (VAE) to provide a reconstruction-based guidance signal. The OCL model learns to extract features that mimic the VAE’s latent representations, which ideally capture the essence of individual objects.

However, a significant bottleneck exists: VAEs are inherently scale-sensitive. They possess a "comfort zone"—a specific range of resolutions where they can represent objects with high fidelity. When an object is too small, the representation becomes noisy; when it is too large, local details are often lost or blurred. Because standard OCL frameworks utilize a single-scale VAE, they frequently fail to provide clear guidance for objects that fall outside this narrow optimal range.

Methodology: Multi-Scale Fusion (MSF)

To address the scale-sensitivity of VAEs, the authors propose Multi-Scale Fusion (MSF). This approach broadens the model's expertise by processing the scene at multiple resolutions simultaneously.

Overview of the Multi-Scale Fusion architecture, showing the image pyramid, multi-scale VAE encoding, and the resulting augmented feature maps alongside the "comfort zone" hypothesis graph.

The MSF framework consists of three core pillars:

  1. Image Pyramid: The input image is resized into a pyramid of three scales ($N=3$). This ensures that any given object, regardless of its physical size in the original image, will likely appear within the VAE's "comfort zone" at at least one of these scales.
  2. Inter-Scale Fusion (Scale-Invariance): This component focuses on identifying "what" an object is. By using a shared codebook across resolutions, the model can recognize the same object type at different scales. If an object is poorly represented at one scale but clear at another, MSF "borrows" the high-quality features from the superior scale to augment the weaker one.
  3. Intra-Scale Fusion (Scale-Variance): While scale-invariance helps with identification, distinguishing between different instances of the same object type requires preserving unique local details. MSF employs scale-specific codebooks to capture these nuances, ensuring that the final fused representation is both semantically rich and spatially precise.

The "Comfort Zone" Hypothesis

A key conceptual contribution of this work is the formalization of the "Comfort Zone" hypothesis. The researchers demonstrate that neural networks do not perform uniformly across scales. By forcing the model to integrate information from multiple resolutions, MSF ensures that the guidance signal provided to the OCL slots is always derived from a "high-confidence" reference point. This selective augmentation strategy differs from traditional Feature Pyramid Networks (FPNs), which often mix features indiscriminately. Instead, MSF uses codebook matching probabilities to intelligently select the best features for each region.

Experimental Results and Evaluation

The efficacy of MSF was tested across a wide array of datasets, ranging from controlled synthetic environments to unconstrained real-world scenes.

Qualitative comparison of segmentation results across various datasets (ClevrTex, COCO, VOC, MOVi). The "+msf" row demonstrates significantly improved object discovery and boundary definition compared to baseline methods.

Key findings include:

  • Performance Gains: MSF acts as a plug-and-play improver for various OCL architectures. For instance, it improved the Foreground Adjusted Rand Index ($ARI_{fg}$) of the Transformer-based SLATE model by 12% on the ClevrTex dataset. It also boosted the state-of-the-art SlotDiffusion model by 3% ARI on the challenging VOC dataset.
  • Real-World Robustness: Consistent improvements were observed on COCO and VOC, where MSF outperformed other recent enhancement techniques like SysBinder and GDR.
  • Computational Efficiency: Despite the multi-scale processing, the authors maintained efficiency through an invertible projection design that reduced codebook parameters by approximately 81% compared to naive multi-scale baselines. The resulting pre-training time increase was a manageable 31%.

Conclusion and Future Outlook

Multi-Scale Fusion represents a significant step forward in making Object-Centric Learning more robust to the complexities of real-world visual data. By acknowledging and addressing the inherent scale limitations of VAE-based guidance, MSF provides a cleaner, more separable signal for object discovery. While performance gains on highly complex real-world scenes are slightly more modest than on synthetic data, the framework provides a solid foundation for future research. Potential directions include integrating MSF directly into the primary Slot Attention mechanism or applying these multi-scale representations to enhance generative scene modeling.

Summary updated

BibTeX
@inproceedings{zhao2025multi,
  author = {Rongzhen Zhao and Vivienne Wang and Juho Kannala and Joni Pajarinen},
  title = {{Multi-Scale Fusion for Object Representation}},
  booktitle = {International Conference on Learning Representations},
  year = {2025},
  url = {https://openreview.net/forum?id=nobDw4d1k7},
}
Figure