Contrastive Coding for Active Learning under Class Distribution Mismatch

Pan Du, Suyun Zhao, Hui Chen, Shuwen Chai, Hong Chen, Cuiping Li

Abstract

Standard Active Learning (AL) often fails in real-world scenarios where the unlabeled data pool contains "distractor" classes not present in the labeled training set. This paper introduces CCAL (Contrastive Coding based Active Learning), a framework that leverages contrastive learning to filter out irrelevant categories while simultaneously identifying the most informative samples from the target distribution.

The Challenge: Class Distribution Mismatch

Active Learning aims to minimize annotation costs by selecting the most informative data points. However, most AL algorithms assume the unlabeled pool shares the same category distribution as the labeled data. In practice, unlabeled pools often contain "Out-of-Distribution" (OOD) distractors. Standard uncertainty-based AL methods frequently mistake these distractors for highly informative samples, leading to wasted annotation budget—a phenomenon known as Class Distribution Mismatch.

Class distribution mismatch: Unlabeled data often contains 'distractor' classes (OOD_C) not present in the labeled set (ID_C).

Methodology: The CCAL Framework

CCAL addresses this mismatch by learning two distinct types of features through contrastive learning tasks:

  1. Semantic Features ($S_{sem}$): Using a SimCLR-style approach, the model learns category-level representations. This allows the system to calculate a "semantic score" to determine if an unlabeled image belongs to the known labeled categories. Low-scoring samples are identified as OOD distractors and filtered.
  2. Distinctive Features ($S_{dis}$): To identify the most informative samples within the target categories, the model employs a granular contrastive task (e.g., rotation transforms). This identifies unique, representative samples that help refine the model's decision boundaries.
CCAL workflow: Unlabeled samples are scored based on semantic similarity to labeled data and distinctive feature importance to generate a final query score.

The Joint Query Strategy: These scores are combined using a non-linear mapping (specifically a $tanh$ function). This creates a "gatekeeper" mechanism that suppresses OOD samples while highlighting distinctive In-Distribution (ID) samples for labeling.

Experimental Evaluation

The authors evaluated CCAL against several benchmarks (CIFAR10, CIFAR100, and an Artificial Cross-Dataset) with mismatch ratios ranging from 20% to 80%.

Key Results

  • Superior Robustness: CCAL consistently outperformed state-of-the-art methods like VAAL, Coreset, and BADGE. As the data became "dirtier" (higher mismatch), the performance gap widened.
  • Efficiency Gains: On CIFAR100 with an 80% mismatch ratio, CCAL reached supervised-level accuracy using only 26.09% of the total data, significantly outperforming random sampling by approximately 7.6%.
  • Dual-Error Minimization: By decomposing AL error into "Invalid Query Error" (OOD waste) and "Valid Query Error" (uninformative ID selection), CCAL is the first framework to theoretically and practically minimize both.

Conclusion and Future Directions

CCAL provides a robust solution for Active Learning in messy, real-world data environments. While currently focused on computer vision, future work includes extending the framework to Natural Language Processing (NLP) and automating hyperparameter selection for the joint query strategy. Despite the computational cost of training dual contrastive encoders, the reduction in human annotation costs makes CCAL a highly practical tool for industrial machine learning pipelines.

Summary updated

Abstract

Active learning (AL) is successful based on the assumption that labeled and unlabeled data are obtained from the same class distribution. However, its performance deteriorates under class distribution mismatch, wherein the un-labeled data contain many samples out of the class distribution of labeled data. To effectively handle the problems under class distribution mismatch, we propose a contrastive coding based AL framework named CCAL. Unlike the existing AL methods that focus on selecting the most informative samples for annotating, CCAL extracts both semantic and distinctive features by contrastive learning and combines them in a query strategy to choose the most informative un-labeled samples with matched categories. Theoretically, we prove that the AL error of CCAL has a tight upper bound. Experimentally, we evaluate its performance on CIFAR10, CIFAR100, and an artificial cross-dataset that consists of five datasets; consequently, CCAL achieves state-of-the-art performance by a large margin with remarkably lower annotation cost. To the best of our knowledge, CCAL is the first work related to AL for class distribution mismatch.
BibTeX
@inproceedings{du2021contrastive,
  author = {Pan Du and Suyun Zhao and Hui Chen and Shuwen Chai and Hong Chen and Cuiping Li},
  title = {{Contrastive Coding for Active Learning under Class Distribution Mismatch}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  doi = {10.1109/iccv48922.2021.00880},
}
Figure