Unsupervised Multi-Object Segmentation Using Attention and Soft-Argmax

Bruno Sauvalle, Arnaud de La FortelleWACV 2023

TL;DR

This paper introduces a novel unsupervised approach for learning object-centric representations, enabling simultaneous object detection and segmentation without relying on predefined spatial priors. By leveraging translation-equivariant attention and a soft-argmax mechanism, the model achieves state-of-the-art performance on complex synthetic benchmarks, paving the way for more robust visual understanding in real-world scenarios.

What Problem Does This Paper Address?

Current methods for unsupervised object-centric representation learning often struggle with complex, real-world scenes, performing well on simplified synthetic datasets but failing to generalize. A key challenge is to disentangle individual objects from their backgrounds and from each other, allowing for a structured representation where each object is associated with its own feature vector and spatial location. Existing approaches often rely on predefined spatial grids or anchors, limiting their flexibility. This paper aims to overcome these limitations by developing an architecture that can robustly detect and segment multiple objects in complex scenes without any prior assumptions about their spatial arrangement.

Methods

The proposed architecture, dubbed AST-Seg (Attention and Soft-argmax with Transformer using Segformer), comprises two main components: a background model and a foreground model.

  • Background Model: A deterministic convolutional autoencoder is used to learn the low-dimensional manifold of background images.
  • Foreground Model: This is the core of the novel contribution. It uses a high-resolution feature map generator (specifically, the Segformer model) to produce a feature map and multiple scalar attention logit maps, one for each potential object. These attention maps are then normalized using a softmax operator to create attention maps.
  • Object Localization: The soft-argmax operator is applied to these attention maps to predict the coordinates of detected objects. The paper provides a theoretical justification for using soft-argmax as an equivariant localization operator.
  • Object Feature Extraction: Attention maps are also used as weights to extract object-specific feature vectors from the main feature map.
  • Transformer Encoder: A transformer encoder refines these initial object detections (feature vectors and coordinates), handling redundancies and occlusions.
  • Glimpse Generation and Reconstruction: A glimpse generator creates object image layers and masks based on the refined object features. These are then scaled and translated to their predicted positions using a spatial transformer network. The background layer and the generated object layers are combined using learned activation levels and masks to reconstruct the full image.

The training process incorporates a reconstruction loss and a pixel entropy loss, with a curriculum learning strategy to manage the interaction between the background and foreground models.

Datasets

The paper evaluates its model on several public benchmarks:

  • CLEVRTEX: A dataset for unsupervised object instance segmentation.
  • CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning.
  • ShapeStacks: A dataset focused on learning physical intuition for object stacking.
  • ObjectsRoom: A dataset for unsupervised scene decomposition and representation.

Additionally, the model's generalization capabilities are tested on out-of-distribution (OOD) and CAMO datasets, using a model trained solely on CLEVRTEX.

Key Insights and Ideas

  • Theoretical Justification for Soft-Argmax: The paper provides a formal argument for why the soft-argmax operator is a natural choice for equivariant object localization, connecting it to the properties of translation-equivariant operators.
  • Attention as a Unified Mechanism: Attention maps are shown to serve a dual purpose: extracting object features and localizing objects, simplifying the architecture.
  • Transformer for Refinement: The use of a transformer encoder effectively addresses challenges like object occlusion and redundant detections, which are common in unsupervised settings.
  • Curriculum Learning for Stability: The proposed curriculum learning strategy, which involves pre-training the background model, is crucial for achieving stable and improved performance, especially on more complex datasets.
  • Generalization to Unseen Data: The model demonstrates strong generalization capabilities to datasets with unseen textures and shapes, indicating robustness beyond the training distribution.

Key Results

The proposed AST-Seg model, particularly when trained with curriculum learning (AST-Seg-B3-CT), significantly outperforms existing state-of-the-art methods on complex benchmarks like CLEVR and CLEVRTEX. For instance, on CLEVR, it achieves a mIoU of 90.27% and an ARI-FG of 98.26%, a substantial improvement over previous methods. The model also shows impressive generalization to OOD and CAMO datasets, achieving a mIoU of 67.50% and 83.14% respectively, even when trained only on CLEVRTEX. Ablation studies confirm the importance of the transformer encoder for complex datasets and highlight the effectiveness of the attention and soft-argmax mechanisms.

Limitations and Future Work

The authors acknowledge that the model's main limitation is the management of shadows, which can sometimes be misinterpreted as separate objects or integrated into object segmentations. Future work could explore more sophisticated ways to handle such challenging visual phenomena. The paper also suggests that the Segformer backbone can be replaced with other generators, and an ablation study shows that removing the transformer encoder can even improve results on simpler datasets, hinting at potential architectural optimizations for different scene complexities.

Summary updated

BibTeX
@inproceedings{sauvalle2023unsupervised,
  author = {Bruno Sauvalle and Arnaud de La Fortelle},
  title = {{Unsupervised Multi-Object Segmentation Using Attention and Soft-Argmax}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
}
Figure