Toward Unsupervised, Multi-object Discovery in Large-Scale Image Collections

Huy V. Vo, Patrick Pérez, Jean Ponce

Abstract

This paper addresses the problem of discovering the objects present in a collection of images without any supervision. We build on the optimization approach of Vo et al. [34] with several key novelties: (1) We propose a novel saliency-based region proposal algorithm that achieves significantly higher overlap with ground-truth objects than other competitive methods. This procedure leverages off-the-shelf CNN features trained on classification tasks without any bounding box information, but is otherwise unsupervised. (2) We exploit the inherent hierarchical structure of proposals as an effective regularizer for the approach to object discovery of [34], boosting its performance to significantly improve over the state of the art on several standard benchmarks. (3) We adopt a two-stage strategy to select promising proposals using small random sets of images before using the whole image collection to discover the objects it depicts, allowing us to tackle, for the first time, the discovery of multiple objects in each one of the pictures making up datasets with up to 20,000 images, an over five-fold increase compared to existing methods, and a first step toward true large-scale unsupervised image interpretation.

Introduction

Object discovery—the task of localizing objects in unlabeled image collections—remains a fundamental challenge in computer vision. While supervised detection has matured, it is constrained by the need for expensive, manually annotated datasets. Existing unsupervised methods often suffer from three primary limitations: a bias toward single-object images, reliance on "unsupervised" proposals that actually leak supervision (e.g., Selective Search), and poor scalability beyond small, curated datasets. This work introduces a framework that overcomes these hurdles, enabling multi-object discovery at a scale of 20,000 images.

Methodology

The proposed approach extends the Object and Structure Discovery (OSD) framework, which models image collections as graphs where nodes represent images and edges capture shared visual patterns.

Saliency-Based Region Proposals

A core innovation is the generation of region proposals directly from pre-trained CNN feature maps (e.g., VGG16) without bounding box supervision. By aggregating these maps into a global saliency map, the method identifies potential object "seeds" using persistence—a concept from topological data analysis. This allows for the identification of robust local maxima (peaks) that correspond to distinct objects.

Visualization of the saliency-based region proposal process. From left to right: input image, saliency map, identified peaks, and the iterative discovery of multiple objects (bicycle, person, sign) using persistence-based seeds.

Regularized OSD (rOSD)

To address the tendency of original OSD to select multiple redundant boxes for a single prominent object, the authors introduce a structural regularizer. By grouping proposals based on their originating saliency peak, the algorithm is constrained to select at most one box per peak. This "rOSD" formulation naturally encourages the discovery of diverse objects within a single image.

Large-Scale Optimization

To scale to tens of thousands of images, a two-stage strategy is employed. First, a "proxy" discovery phase on random image subsets filters out low-quality proposals. Second, the full optimization is performed on the entire collection using only the refined candidate set, significantly reducing the computational and memory footprint.

Experimental Results

The framework was evaluated on standard benchmarks (Object Discovery, PASCAL VOC 2007/2012) and a new large-scale dataset, COCO_20k.

  • Localization Performance: On VOC_all, rOSD achieved a 49.3% discovery rate, outperforming the original OSD (40.7%) and other state-of-the-art methods like DDT+ (43.4%).
  • Multi-Object Discovery: In complex scenes (VOC12), rOSD reached 40.4%, a 10-point improvement over previous benchmarks.
  • Scalability: The method successfully processed the COCO_20k dataset (19,817 images), demonstrating a five-fold increase in scale over prior optimization-based discovery methods.

Discussion and Conclusion

This work demonstrates that CNNs trained for classification learn powerful, category-independent spatial priors that can be harnessed for unsupervised discovery. By integrating topological persistence and structural regularization, the framework effectively handles multi-object scenes and large-scale data.

Future work may address the computational intensity (currently ~478 CPU hours for 3,550 images) and explore "pure" unsupervised features from self-supervised models like MoCo or SimCLR to remove the reliance on ImageNet pre-training. Additionally, resolving cases where objects overlap significantly in the saliency map remains an open challenge.

Summary updated

Abstract

This paper addresses the problem of discovering the objects present in a collection of images without any supervision. We build on the optimization approach of Vo et al. [34] with several key novelties: (1) We propose a novel saliency-based region proposal algorithm that achieves significantly higher overlap with ground-truth objects than other competitive methods. This procedure leverages off-the-shelf CNN features trained on classification tasks without any bounding box information, but is otherwise unsupervised. (2) We exploit the inherent hierarchical structure of proposals as an effective regularizer for the approach to object discovery of [34], boosting its performance to significantly improve over the state of the art on several standard benchmarks. (3) We adopt a two-stage strategy to select promising proposals using small random sets of images before using the whole image collection to discover the objects it depicts, allowing us to tackle, for the first time (to the best of our knowledge), the discovery of multiple objects in each one of the pictures making up datasets with up to 20,000 images, an over five-fold increase compared to existing methods, and a first step toward true large-scale unsupervised image interpretation.
BibTeX
@inproceedings{vo2020toward,
  author = {Huy V. Vo and Patrick Pérez and Jean Ponce},
  title = {{Toward Unsupervised, Multi-object Discovery in Large-Scale Image Collections}},
  booktitle = {European Conference on Computer Vision},
  year = {2020},
  url = {https://openreview.net/forum?id=UtkCadotmLG},
}
Figure