Beyond Learning Features: Training a Fully-Functional Classifier with ZERO Instance-Level Labels

Deepak Babu Sam, Abhinav Agarwalla, Venkatesh Babu Radhakrishnan

Introduction: The Labeling Bottleneck

In modern deep learning, the term "unsupervised" often masks a reliance on labels at the final stage. While self-supervised methods excel at learning rich feature representations, they typically require a small set of labeled examples to map these features to specific classes (e.g., mapping a cluster to the label "Dog"). This instance-level labeling is expensive, difficult to scale, and limits the adaptability of models to new, unlabeled data streams.

This paper proposes a paradigm shift: training a classifier with zero instance-level labels. Instead of explicit image-label pairs, the framework leverages the inherent semantic structure of category names to guide the learning process.

Comparison of learning paradigms: Unsupervised Clustering and Self-supervised Learning vs. the proposed approach which removes the need for instance-level labels.

Methodology: Bridging Vision and Semantics

The proposed framework operates in three distinct stages to transform raw images into a task-specific classifier without human intervention.

1. Self-supervised Feature Learning (SFL)

The process begins by training a backbone network ($f_{BB}$) using standard contrastive learning techniques (such as SimCLR or MoCoV2). This stage allows the model to learn visual primitives and object-level features from unlabeled images.

2. Feature Clustering (FC)

The learned features are then grouped into clusters. While these clusters lack semantic names at this stage, they provide a visual grouping of similar objects, serving as a structural foundation for the classifier.

3. Cluster-Guided Distribution Matching

This is the core innovation of the paper. The authors align visual clusters with semantic categories using two key components:

  • Semantic Prior: By using Word2Vec embeddings of class names, the model calculates semantic similarities between categories. This creates a "prior" distribution that reflects how classes relate to one another (e.g., 'cat' is closer to 'dog' than to 'car').
  • Optimal Transport (Sinkhorn Loss): The model treats the classifier's predictions and the semantic prior as two distributions. It employs the Sinkhorn algorithm to efficiently align these distributions, effectively "matching" visual clusters to their most likely semantic labels.
  • Cluster Guidance: To maintain consistency, the visual clusters from Stage 2 act as a regularizer, ensuring that images within the same visual cluster are assigned to the same semantic category.
Detailed view of the proposed approach: matching visual representations to semantic priors using distribution matching.

Experimental Results

The authors evaluated the zero-label classifier across both nominal and ordinal classification tasks.

Nominal Classification

On standard datasets like CIFAR-10 and ImageNet-10, the method demonstrated significant capabilities:

  • CIFAR-10: Achieved 42.1% accuracy, far exceeding the random baseline (10%) and models using random classification layers (11.2%).
  • ImageNet-10: Reached 47.3% accuracy (baseline 6.6%).

Ordinal Classification

The method is particularly effective for ordinal tasks where labels have a strict linear relationship (e.g., age estimation or disease severity).

  • Aesthetic Rating: On the Aesthetic dataset, it achieved 57.93% accuracy, approaching the performance of supervised methods (69-72%).
  • Medical Imaging: The model was successfully applied to Diabetic Retinopathy (DR) grading, showing its potential in specialized domains.

Key Insights

  • Semantic Structure as Supervision: The semantic relationships between words (e.g., "Dog" vs "Cat") provide a "free" supervisory signal that can replace instance-level labels.
  • Distribution Alignment: Rather than individual mapping, the model learns to match the global "shape" of the visual data to the "shape" of the semantic labels.
  • Visual-Semantic Synergy: The alignment of self-supervised visual clusters with Word2Vec priors proves to be a robust alternative to traditional supervised fine-tuning.

Limitations and Future Directions

While promising, the approach faces challenges in scaling to datasets with thousands of categories, where semantic signatures become increasingly crowded. Additionally, the current formulation assumes a relatively balanced class distribution. Future work will focus on handling class imbalance and improving the resolution of semantic priors to narrow the gap with fully supervised models.

Summary updated

Abstract

We attempt to train deep neural networks for classification without using any labeled data. Existing unsupervised methods, though mine useful clusters or features, require some annotated samples to facilitate the final task-specific predictions. This defeats the true purpose of unsupervised learning and hence we envisage a paradigm of `true' self-supervision, where absolutely no annotated instances are used for training a classifier. The proposed method first pretrains a deep network through self-supervision and performs clustering on the learned features. A classifier layer is then appended to the self-supervised network and is trained by matching the distribution of the predictions to that of a predefined prior. This approach leverages the distribution of labels for supervisory signals and consequently, no image-label pair is needed. Experiments reveal that the method works on major nominal as well as ordinal classification datasets and delivers significant performance.
BibTeX
@inproceedings{sam2022beyond,
  author = {Deepak Babu Sam and Abhinav Agarwalla and Venkatesh Babu Radhakrishnan},
  title = {{Beyond Learning Features: Training a Fully-Functional Classifier with ZERO Instance-Level Labels}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.1609/aaai.v36i2.20113},
}

Loading related content...

Figure