Toward unsupervised, multi-object discovery in large-scale image collections

Huy V. Vo, Patrick Pérez, Jean Ponce

Abstract

This paper addresses the problem of discovering the objects present in a collection of images without any supervision. We build on the optimization approach of Vo et al. (CVPR'19) with several key novelties: (1) We propose a novel saliency-based region proposal algorithm that achieves significantly higher overlap with ground-truth objects than other competitive methods. This procedure leverages off-the-shelf CNN features trained on classification tasks without any bounding box information, but is otherwise unsupervised. (2) We exploit the inherent hierarchical structure of proposals as an effective regularizer for the approach to object discovery of Vo et al., boosting its performance to significantly improve over the state of the art on several standard benchmarks. (3) We adopt a two-stage strategy to select promising proposals using small random sets of images before using the whole image collection to discover the objects it depicts, allowing us to tackle, for the first time, the discovery of multiple objects in each one of the pictures making up datasets with up to 20,000 images, an over five-fold increase compared to existing methods.

Introduction

In computer vision, unsupervised object discovery—the ability for a machine to identify and locate objects in unlabeled image collections—remains a significant challenge. While supervised object detection has seen massive success, it relies on expensive, manually annotated datasets. Previous unsupervised methods have largely been limited by two factors: the "single-object trap," where algorithms only identify the most prominent item in a scene, and the "scaling wall," where memory-intensive optimization prevents application to large datasets.

This research introduces a framework that overcomes these hurdles, enabling the discovery of multiple objects in datasets containing up to 20,000 images—a five-fold increase over previous state-of-the-art benchmarks.

Methodology

The proposed framework introduces three critical innovations to the Object and Structure Discovery (OSD) paradigm:

Unsupervised Saliency Proposals

Instead of relying on supervised region proposal methods, the authors utilize off-the-shelf CNN features (e.g., VGG16) to generate saliency maps. To distinguish between distinct objects, they apply persistence—a concept from topological data analysis—to identify robust local maxima in these maps. This allows the model to treat the saliency map like a topographical landscape, identifying "prominent peaks" that correspond to individual objects rather than noise.

Visualization of the saliency-based proposal generation and iterative multi-object discovery process.

Regularized OSD (rOSD)

Standard OSD often suffers from redundancy, selecting multiple overlapping boxes for the same object. The authors introduce a structural constraint: the algorithm is restricted to picking only one proposal from each hierarchical group identified by the saliency method. This regularization forces the model to look for diverse objects rather than redundant views of the same primary subject.

Two-Stage Scaling Strategy

To manage the computational load of 20,000 images, a "divide and conquer" approach is used. A proxy version of the algorithm runs on small random subsets to filter for high-potential proposals. The full optimization is then executed on the entire collection using only these refined candidates.

Experiments and Results

The method was evaluated across several benchmarks, including a new large-scale dataset, COCO_20k, featuring nearly 20,000 images and 143,000 objects.

  • State-of-the-Art Performance: On the VOC_all dataset, the rOSD method achieved a 49.3% discovery rate, significantly outperforming previous leaders (40.7% and 43.4%).
  • Multi-Object Discovery: The lead was even more pronounced in multi-object settings. On VOC12, the method achieved 40.4% accuracy, compared to 30.3% for the nearest competitor.
  • Scalability: The two-stage strategy successfully maintained high performance on the COCO_20k dataset, proving that unsupervised discovery can scale to realistic, large-scale image collections.

Discussion and Limitations

While this work represents a major step toward true large-scale unsupervised image interpretation, some challenges remain. The computational cost is still high, requiring significant CPU and GPU resources for large datasets. Furthermore, while the discovery process is unsupervised, the underlying CNN features were pre-trained on ImageNet (a supervised task). Future work may explore using features trained via self-supervised learning to achieve a "purely" unsupervised pipeline. Finally, the algorithm still faces difficulties in highly crowded scenes where objects overlap significantly, sometimes merging them into a single discovery box.

Summary updated

Abstract

This paper addresses the problem of discovering the objects present in a collection of images without any supervision. We build on the optimization approach of Vo et al. (CVPR'19) with several key novelties: (1) We propose a novel saliency-based region proposal algorithm that achieves significantly higher overlap with ground-truth objects than other competitive methods. This procedure leverages off-the-shelf CNN features trained on classification tasks without any bounding box information, but is otherwise unsupervised. (2) We exploit the inherent hierarchical structure of proposals as an effective regularizer for the approach to object discovery of Vo et al., boosting its performance to significantly improve over the state of the art on several standard benchmarks. (3) We adopt a two-stage strategy to select promising proposals using small random sets of images before using the whole image collection to discover the objects it depicts, allowing us to tackle, for the first time (to the best of our knowledge), the discovery of multiple objects in each one of the pictures making up datasets with up to 20,000 images, an over five-fold increase compared to existing methods, and a first step toward true large-scale unsupervised image interpretation.
BibTeX
@inproceedings{vo2020toward,
  author = {Huy V. Vo and Patrick Pérez and Jean Ponce},
  title = {{Toward unsupervised, multi-object discovery in large-scale image collections}},
  booktitle = {European Conference on Computer Vision},
  year = {2020},
  url = {https://openreview.net/forum?id=UtkCadotmLG},
}
Figure