Contrast-Enhanced Semi-supervised Text Classification with Few Labels

Austin Cheng-Yun Tsai, Sheng-Ya Lin, Li-Chen FuAAAI 2022 · DOI

Abstract

Deep learning models for text classification typically require vast amounts of labeled data, which is often unavailable in specialized domains. We present the Contrast-Enhanced Semi-supervised Text Classification (CEST) framework, which achieves high performance using as few as 30 labeled examples per class. CEST integrates Bayesian uncertainty estimates for sample selection, a reliability-aware similarity graph for contrastive learning, and noise-robust loss functions to mitigate confirmation bias. Our results demonstrate that CEST outperforms existing state-of-the-art methods without the need for resource-intensive data augmentation techniques like back-translation.

Introduction

Modern NLP relies heavily on large-scale labeled datasets, which are often prohibitively expensive to obtain in niche fields like medical diagnosis or legal analysis. Semi-supervised learning (SSL) offers a solution by utilizing a small amount of labeled data alongside a large volume of unlabeled data. However, traditional SSL methods often suffer from high computational costs (e.g., requiring Neural Machine Translation for augmentation) and "confirmation bias," where models reinforce their own pseudo-labeling errors. CEST addresses these challenges through a robust, three-pronged architecture designed for data efficiency.

The CEST Framework

The CEST framework optimizes the self-training loop by focusing on sample quality and relationship modeling across five key steps, as illustrated in Figure 1.

The CEST framework: (1) Pseudo-annotation, (2) Certainty estimation, (3) Sample selection, (4) Training with noise-robust loss and graph regularization, and (5) Model update.

Certainty-Driven Sample Selection

Instead of relying on raw confidence scores, which can be misleading, CEST employs a Bayesian Neural Network approach via Monte-Carlo Dropout. By executing multiple forward passes with different neurons deactivated, the system calculates Information Gain. This allows the model to identify samples that are both likely correct and highly informative for the model's learning trajectory, balancing certainty with exploration.

Graph-based Contrast Induction

To enforce the "smoothness assumption"—that similar inputs should share similar labels—CEST constructs a similarity graph where nodes represent data points. Edges are pruned based on predictive variance to ensure only reliable relationships are modeled. A contrastive loss then pulls similar points together and pushes dissimilar ones apart in the feature space, creating clearer boundaries between classes.

Noise-Robust Learning

Recognizing that pseudo-labels are inherently noisy, CEST utilizes Partially Huberised Cross-Entropy (PHCE). This loss function linearizes the gradient for high-loss (difficult) samples, preventing the model from aggressively memorizing what are likely to be incorrect pseudo-labels and thus mitigating confirmation bias.

Experimental Evaluation

CEST was evaluated on five standard NLP benchmarks: AG News and DBpedia (Topic Classification), and IMDB, Elec, and SST-2 (Sentiment Analysis). The experiments focused on the "few-label" regime, using only 30 labeled samples per class for training and validation—a tiny fraction of the thousands usually required for these datasets.

Key Results

CEST achieved an average accuracy of 91.57% across all datasets, outperforming the previous state-of-the-art (UST) by 2.11% and the standard BERT baseline by nearly 8%. Remarkably, with only 30 labels per class, CEST reached within 3.04% of the performance of fully supervised models trained on the entire dataset.

Accuracy improvement across various datasets over ten self-training iterations. Solid lines represent CEST performance.

As shown in Figure 2, the model's accuracy improves consistently over self-training iterations. Even when pushed to the extreme of only 10 labels per class, CEST maintained a significant lead over other methods, proving its high data efficiency and independence from external tools like Neural Machine Translation.

Conclusion

CEST provides a highly efficient path for text classification in low-resource settings. By removing the dependency on external NMT systems and addressing confirmation bias through noise-robust loss and graph regularization, it sets a new standard for data-efficient NLP. Future work will explore domain-specific applications in healthcare and legal analysis where data scarcity is most prevalent.

Summary updated

BibTeX
@inproceedings{tsai2022contrast,
  author = {Austin Cheng-Yun Tsai and Sheng-Ya Lin and Li-Chen Fu},
  title = {{Contrast-Enhanced Semi-supervised Text Classification with Few Labels}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.1609/aaai.v36i10.21391},
}
Figure