Abstract
Unsupervised image classification aims to categorize images without human-provided labels. Traditional end-to-end methods often struggle with "low-level bias," grouping images by color or texture rather than semantic content. This paper introduces SCAN (Semantic Clustering by Adopting Nearest neighbors), a three-stage framework that decouples feature learning from clustering. By leveraging self-supervised representations and enforcing consistency among semantic neighbors, SCAN achieves significant performance gains, reaching 88.3% accuracy on CIFAR10 and 80.9% on STL10, setting a new state-of-the-art for unsupervised classification.
Introduction
The challenge of unsupervised computer vision is to automatically organize unlabeled images into meaningful categories. While supervised learning has seen immense success, unsupervised methods often fall into "degenerate solutions" or focus on superficial visual cues. SCAN addresses these issues by separating the representation learning phase from the clustering phase, ensuring that the model focuses on high-level semantic features.
Methodology
The SCAN framework consists of three distinct stages:
1. Self-Supervised Pretext Task
The model first learns a feature representation using a self-supervised task like instance discrimination (e.g., SimCLR). This stage produces an embedding space where images are clustered based on visual similarity without any label guidance.
2. Semantic Clustering (SCAN)
The core of the method is the Semantic Clustering stage. For each image, the model identifies its $K$ nearest neighbors in the pre-trained feature space. The key insight is that in a well-trained self-supervised space, these neighbors are highly likely to belong to the same semantic class.
As shown in the figure above, the nearest neighbors (columns 2-5) for a query image (column 1) consistently share semantic categories like turtles, musical instruments, or fish. To quantify this, the authors analyzed the percentage of "correct pairs" (neighbors sharing the same ground-truth label) across different datasets and values of $K$.
The clustering head is then trained to maximize consistency between an image and its neighbors while using an entropy term to ensure a balanced distribution across clusters.
3. Self-Labeling
In the final stage, the model identifies high-confidence samples and treats their cluster assignments as pseudo-labels. The network is then fine-tuned using standard cross-entropy loss on strongly augmented versions of these samples to further refine the classification boundaries.
Key Results
SCAN demonstrates a massive leap over previous unsupervised methods. On CIFAR10, it achieved 88.3% accuracy, a 26.6% improvement over the previous best (IIC). Similar gains were observed on STL10 (80.9%) and CIFAR100-20 (50.7%). On ImageNet, SCAN produced semantically coherent clusters and even outperformed some semi-supervised methods that utilized 1% of labels.
Discussion and Conclusion
The success of SCAN highlights that decoupling feature learning from clustering is essential for avoiding low-level biases. By using nearest neighbors as semantic proxies, the model gains a powerful signal for grouping. While the method is robust to overclustering, future work may focus on automatically determining the optimal number of clusters and resolving ambiguities in superclass definitions.
Summary updated