Unsupervised High-level Feature Learning by Ensemble Projection for Semi-supervised Image Classification and Image Clustering

Dengxin Dai, Luc Van GoolarXiv 2016 · arXiv

Introduction

In modern computer vision, the disparity between available data and annotated labels is a significant bottleneck. While deep learning has achieved remarkable success in image classification, its performance is heavily dependent on massive, manually labeled datasets. Semi-supervised learning (SSL) aims to mitigate this by leveraging a small amount of labeled data alongside a vast pool of unlabeled images. However, many traditional SSL methods focus on regularizing classifiers over data graphs, which can be sensitive to the underlying data structure.

This paper proposes a different approach: Ensemble Projection (EP). Instead of merely refining a classifier, EP uses unlabeled data to learn a high-level, dataset-specific feature representation. By creating and learning from "surrogate" classes, the method transforms raw features into a more discriminative space, significantly improving performance in both classification and clustering tasks.

The Ensemble Projection Framework

The EP framework operates on the principle of learning from automatically generated labels. The process is divided into three primary stages: sampling, discriminative learning, and projection.

Overview of the Ensemble Projection (EP) framework showing sampling, training, and projection stages.

1. Max-Min Sampling

To learn features without ground-truth labels, the system must create "surrogate classes." This is achieved through two key assumptions:

  • Local-consistency: Images that are very close in the initial feature space likely belong to the same class.
  • Exotic-consistency: Images that are very far apart almost certainly belong to different classes.

The algorithm uses Max-Min Sampling to identify a "skeleton" of images that are maximally distant from each other (the Max-step). It then clusters the nearest neighbors of these skeleton images to form compact, distinct prototypes (the Min-step). Each prototype serves as a surrogate class.

2. Discriminative Learning

Once the surrogate classes are established, the system treats them as ground truth for a classification task. A simple discriminative classifier, such as Logistic Regression, is trained to distinguish between these prototypes. In doing so, the model is forced to learn high-level visual attributes that separate these automatically discovered concepts.

3. Ensemble Projection

Because a single set of random prototypes may be noisy or incomplete, the authors employ an ensemble strategy. The sampling and training process is repeated $T$ times. For any given image, its final representation—the Projection Vector—is the concatenation of its classification scores across all $T$ surrogate classifiers. This high-dimensional vector represents the image's similarity to hundreds of diverse visual concepts discovered across the unlabeled dataset.

Robustness and the "Wisdom of the Crowds"

A critical finding of the research is the robustness of the ensemble approach to label noise. Even if the surrogate classes are imperfect (containing "noise" where images from different true classes are grouped together), the ensemble effect cancels out these errors as the number of trials ($T$) increases.

Impact of ensemble size (T) and label noise (R) on classification precision.

As shown in the experimental results, even with a noise level ($R$) as high as 80% in the surrogate sets, the classification precision eventually converges to a high level as $T$ reaches 1,000 to 2,000. This "wisdom of the crowds" allows the model to extract reliable semantic features from inherently noisy, unsupervised data.

Key Results and Performance

The authors validated EP across eight diverse benchmarks, including object recognition (Caltech-101), scene recognition (Indoor-67), and specialized tasks like satellite imagery (LandUse-21).

  • Superior to SSL Standards: EP consistently outperformed established semi-supervised methods like Harmonic Function (HF) and Anchor Graph (AG). On the Indoor-67 dataset with only 5 labels per class, EP improved precision by 11.9% over its closest competitor.
  • Enhancing Deep Learning: One of the most powerful applications of EP is as a "feature tuner." When applied on top of pre-trained CNN features, it adapts those general representations to the specific patterns of the target dataset. On Scene-15, this raised baseline k-NN accuracy from 62.4% to 75.6%.
  • Unsupervised Clustering: In purely unsupervised scenarios, EP significantly improved the "purity" of image clusters. For the Event-8 dataset, it boosted k-means purity by 9.6%.

Conclusion

Ensemble Projection offers a robust and efficient way to bridge the gap between unsupervised feature learning and supervised classification. By treating unlabeled data as a source of surrogate concepts, EP learns high-level representations that are both discriminative and dataset-specific. While the resulting feature vectors are high-dimensional, the method's parallelizability and significant performance gains make it a compelling tool for modern computer vision pipelines.

Summary updated

Abstract

This paper investigates the problem of image classification with limited or no annotations, but abundant unlabeled data. The setting exists in many tasks such as semi-supervised image classification, image clustering, and image retrieval. Unlike previous methods, which develop or learn sophisticated regularizers for classifiers, our method learns a new image representation by exploiting the distribution patterns of all available data for the task at hand. Particularly, a rich set of visual prototypes are sampled from all available data, and are taken as surrogate classes to train discriminative classifiers; images are projected via the classifiers; the projected values, similarities to the prototypes, are stacked to build the new feature vector. The training set is noisy. Hence, in the spirit of ensemble learning we create a set of such training sets which are all diverse, leading to diverse classifiers. The method is dubbed Ensemble Projection (EP). EP captures not only the characteristics of individual images, but also the relationships among images. It is conceptually simple and computationally efficient, yet effective and flexible. Experiments on eight standard datasets show that: (1) EP outperforms previous methods for semi-supervised image classification; (2) EP produces promising results for self-taught image classification, where unlabeled samples are a random collection of images rather than being from the same distribution as the labeled ones; and (3) EP improves over the original features for image clustering. The code of the method is available on the project page.
BibTeX
@misc{dai2016unsupervised,
  author = {Dengxin Dai and Luc Van Gool},
  title = {{Unsupervised High-level Feature Learning by Ensemble Projection for Semi-supervised Image Classification and Image Clustering}},
  year = {2016},
  eprint = {1602.00955},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1602.00955},
}
Figure