Image Segmentation-Based Unsupervised Multiple Objects Discovery

Sandra Kara, Hejer Ammar, Florian Chabot, Quoc-Cuong PhamWACV 2023

TL;DR

This paper introduces a novel, fully unsupervised approach for discovering and segmenting multiple objects within images. By leveraging self-supervised Vision Transformer (SS-ViT) features and a two-stage process of intra-image part discovery followed by dataset-level semantic classification, the method achieves state-of-the-art results in unsupervised object detection and segmentation without requiring any labeled data.

What Problem Does This Paper Address?

Training deep learning models for computer vision tasks like object detection and segmentation typically requires massive amounts of meticulously labeled data. This annotation process is costly and time-consuming. While semi-supervised and weakly supervised methods aim to reduce this burden, this paper tackles the even more challenging problem of Unsupervised Object Discovery (UOD). UOD aims to identify and localize objects in images without any prior human supervision. Existing UOD methods often rely on inter-image similarities, which can be computationally expensive and dependent on supervised features. Furthermore, many methods struggle with discovering multiple objects within a single image or are limited by noise in the proposed object candidates. This work aims to overcome these limitations by proposing a robust, fully unsupervised, bottom-up approach for discovering multiple object instances.

Methods

The proposed method employs a two-stage, bottom-up strategy:

  1. Intra-image Analysis: Object Parts Segmentation: This stage focuses on discovering local semantic concepts (object parts) within individual images. It extends the TokenCut approach by using self-supervised Vision Transformer (SS-ViT) features (specifically from DINO). Instead of just using the second eigenvector for spectral clustering, they utilize N eigenvectors to capture multiple objects. A key innovation here is an algorithm to automatically determine an optimal number of clusters (K) without prior knowledge of the number of objects, aiming to discover multiple object parts while limiting over-segmentation.
  2. Dataset-level Analysis: Self-supervised Classification: This stage leverages the entire dataset to learn semantic information.
  3. Object Proposal Re-ranking: Selective Search (SeSe) is used to generate initial object proposals. These proposals are then re-ranked using a novel objectness score that combines local similarity (how similar a proposal is to overlapping neighbors), global dissimilarity (how dissimilar a proposal is to non-overlapping regions), and entropy (to penalize homogeneous, likely background regions). This re-ranking aims to reduce noise and select more reliable proposals.
  4. Training Classifiers: The top P re-ranked proposals are used to generate pseudo-labels for training two CNN-based classifiers:
  5. A multi-class classifier trained on foreground (object) clusters to assign semantic class labels to objects and their parts.
  6. A binary foreground/background classifier trained on all discovered clusters to distinguish objects from background regions.
  7. Instance Segmentation using Dataset-level Information: The trained classifiers are then used to refine the results from the intra-image analysis. The multi-class classifier merges object parts into complete object instances. The foreground/background classifier is then applied to remove any remaining background segments. This final step produces instance masks and confidence scores for each discovered object.

Datasets

The paper evaluates its method on the following datasets:

  • PASCAL VOC 2007 trainval
  • PASCAL VOC 2012 trainval
  • COCO20k (a subset of COCO2014 trainval)
  • VOC2012 validation set (for unsupervised image segmentation)

The authors note that they use these standard benchmarks for evaluation and do not introduce new ones.

Key Insights and Ideas

  • SS-ViT Features are Powerful for Unsupervised Discovery: The paper strongly demonstrates that self-supervised features from Vision Transformers (like DINO) inherently capture semantic separation and local-global alignment, making them highly effective for unsupervised object discovery tasks.
  • Bottom-Up Discovery with Dataset-Level Semantics: The two-stage approach, moving from local part discovery within an image to learning global semantic categories across the dataset, is a robust way to handle the complexity of multiple object discovery without supervision.
  • Automatic Cluster Number Determination: The iterative clustering algorithm to find the optimal number of object parts (K) is crucial for adapting to varying image content and avoiding manual tuning.
  • Multi-faceted Objectness Score: The proposed re-ranking score for object proposals, combining local similarity, global dissimilarity, and entropy, is more effective than standard methods in filtering out noisy candidates.
  • Segmentation as a Foundation for Detection: By first generating instance masks and then using them to train classifiers for object detection, the method inherently produces more precise bounding boxes and reduces noise compared to methods relying solely on proposal ranking.

Key Results

  • State-of-the-art Performance: The method achieves state-of-the-art results in unsupervised multiple objects discovery (measured by odAP@50 and odAP@[50-95]), unsupervised class-agnostic object detection (measured by AP@50), and unsupervised image segmentation (measured by mIOU) on challenging benchmarks like PASCAL VOC and COCO.
  • Improved Precision-Recall Trade-off: The generated pseudo-labels offer a better precision-recall trade-off compared to existing methods, particularly for the more demanding odAP@[50-95] metric, indicating more accurate object localization.
  • Superiority on Complex Scenes: On the more challenging COCO20k dataset, the method outperforms both multi-object discovery (MOD) and single object discovery (SOD) methods, demonstrating its effectiveness in complex scenes with multiple objects.
  • Robustness to Dataset Bias: The approach is less susceptible to dataset biases (like PASCAL VOC's tendency for single-object images) compared to SOD methods.

Limitations and Future Work

  • Recall Disadvantage: The paper acknowledges that its approach, being segmentation-based, naturally proposes fewer object candidates per image compared to proposal-ranking methods. This can put it at a disadvantage in recall-focused metrics like odAP.
  • Pseudo-label Initialization: The authors suggest further investigating the use of the generated pseudo-labels as an initial seed in pseudo-labeling approaches for supervised tasks.
  • Noise Handling: They also propose exploring the use of these object candidates with more advanced noise-handling mechanisms.
  • Ablation Study on Eigenvectors: An ablation study (Table 5) indicates that using 3 eigenvectors for intra-image analysis provides a good precision/recall trade-off for both PASCAL VOC and COCO datasets.

Summary updated

BibTeX
@inproceedings{kara2023image,
  author = {Sandra Kara and Hejer Ammar and Florian Chabot and Quoc-Cuong Pham},
  title = {{Image Segmentation-Based Unsupervised Multiple Objects Discovery}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
}
Figure