SCAN: Learning to Classify Images Without Labels

Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis, Marc Proesmans, Luc Van Gool

Abstract

Unsupervised image classification aims to categorize images without human-provided labels. Traditional end-to-end methods often struggle with "low-level bias," grouping images by color or texture rather than semantic content. This paper introduces SCAN (Semantic Clustering by Adopting Nearest neighbors), a three-stage framework that decouples feature learning from clustering. By leveraging self-supervised representations and enforcing consistency among semantic neighbors, SCAN achieves significant performance gains, reaching 88.3% accuracy on CIFAR10 and 80.9% on STL10, setting a new state-of-the-art for unsupervised classification.

Introduction

The challenge of unsupervised computer vision is to automatically organize unlabeled images into meaningful categories. While supervised learning has seen immense success, unsupervised methods often fall into "degenerate solutions" or focus on superficial visual cues. SCAN addresses these issues by separating the representation learning phase from the clustering phase, ensuring that the model focuses on high-level semantic features.

Methodology

The SCAN framework consists of three distinct stages:

1. Self-Supervised Pretext Task

The model first learns a feature representation using a self-supervised task like instance discrimination (e.g., SimCLR). This stage produces an embedding space where images are clustered based on visual similarity without any label guidance.

2. Semantic Clustering (SCAN)

The core of the method is the Semantic Clustering stage. For each image, the model identifies its $K$ nearest neighbors in the pre-trained feature space. The key insight is that in a well-trained self-supervised space, these neighbors are highly likely to belong to the same semantic class.

Visualizing semantic nearest neighbors in the feature space.

As shown in the figure above, the nearest neighbors (columns 2-5) for a query image (column 1) consistently share semantic categories like turtles, musical instruments, or fish. To quantify this, the authors analyzed the percentage of "correct pairs" (neighbors sharing the same ground-truth label) across different datasets and values of $K$.

Percentage of correct semantic pairs across different datasets and neighbor counts.

The clustering head is then trained to maximize consistency between an image and its neighbors while using an entropy term to ensure a balanced distribution across clusters.

3. Self-Labeling

In the final stage, the model identifies high-confidence samples and treats their cluster assignments as pseudo-labels. The network is then fine-tuned using standard cross-entropy loss on strongly augmented versions of these samples to further refine the classification boundaries.

Key Results

SCAN demonstrates a massive leap over previous unsupervised methods. On CIFAR10, it achieved 88.3% accuracy, a 26.6% improvement over the previous best (IIC). Similar gains were observed on STL10 (80.9%) and CIFAR100-20 (50.7%). On ImageNet, SCAN produced semantically coherent clusters and even outperformed some semi-supervised methods that utilized 1% of labels.

Discussion and Conclusion

The success of SCAN highlights that decoupling feature learning from clustering is essential for avoiding low-level biases. By using nearest neighbors as semantic proxies, the model gains a powerful signal for grouping. While the method is robust to overclustering, future work may focus on automatically determining the optimal number of clusters and resolving ambiguities in superclass definitions.

Summary updated

Abstract

Can we automatically group images into semantically meaningful clusters when ground-truth annotations are absent? The task of unsupervised image classification remains an important, and open challenge in computer vision. Several recent approaches have tried to tackle this problem in an end-to-end fashion. In this paper, we deviate from recent works, and advocate a two-step approach where feature learning and clustering are decoupled. First, a self-supervised task from representation learning is employed to obtain semantically meaningful features. Second, we use the obtained features as a prior in a learnable clustering approach. In doing so, we remove the ability for cluster learning to depend on low-level features, which is present in current end-to-end learning approaches. Experimental evaluation shows that we outperform state-of-the-art methods by large margins, in particular +26.6% on CIFAR10, +25.0% on CIFAR100-20 and +21.3% on STL10 in terms of classification accuracy. Furthermore, our method is the first to perform well on a large-scale dataset for image classification. In particular, we obtain promising results on ImageNet, and outperform several semi-supervised learning methods in the low-data regime without the use of any ground-truth annotations. The code is made publicly available at https://github.com/wvangansbeke/Unsupervised-Classification.
BibTeX
@inproceedings{gansbeke2020scan,
  author = {Wouter Van Gansbeke and Simon Vandenhende and Stamatios Georgoulis and Marc Proesmans and Luc Van Gool},
  title = {{SCAN: Learning to Classify Images Without Labels}},
  booktitle = {European Conference on Computer Vision},
  year = {2020},
  doi = {10.1007/978-3-030-58607-2\_16},
}
Figure