Progressive Class Semantic Matching for Semi-supervised Text Classification

Hai-Ming Xu, Lingqiao Liu, Ehsan Abbasnejad

Abstract

Deep learning models for text classification typically require extensive labeled datasets. Semi-supervised learning (SSL) attempts to mitigate this by utilizing unlabeled data. However, many SSL approaches fail to fully exploit the internal semantic knowledge of pre-trained language models (PLMs) like BERT. This paper introduces Progressive Class Semantic Matching (PCM), a framework that treats BERT as more than a feature extractor. PCM utilizes BERT’s inherent ability to match sentences with topic-related words through a dual-classifier system and dynamic class semantic representations. Experimental results show that PCM achieves state-of-the-art performance in extreme low-resource scenarios, such as having only three labeled examples per class.

Introduction

The primary challenge in semi-supervised text classification is effectively leveraging a small amount of labeled data alongside a large pool of unlabeled text. While fine-tuning PLMs has become standard, the authors argue that standard fine-tuning ignores the rich internal knowledge BERT acquires during pre-training tasks like Next Sentence Prediction. BERT is naturally adept at recognizing semantic relationships between sentences and specific "class-related" words. PCM formalizes this matching capability to create a more robust SSL framework.

Methods: The PCM Framework

The PCM model integrates three co-evolving components to improve classification accuracy progressively:

  1. Standard $K$-way Classifier: A traditional layer predicting one of $K$ categories.
  2. Class Semantic Representation (CSR): Instead of abstract indices, classes are represented as average embeddings of related words (e.g., "football" for "Sports").
  3. Matching Classifier: This component calculates the semantic "match" between an input sentence and the CSR of a class.
The architecture of the PCM model, featuring the dual-classifier approach and BERT encoder.

The training process employs a "Progressive" mechanism. Initially, keywords are identified from the small labeled set. During training on unlabeled data (using Unsupervised Data Augmentation), pseudo-labels are only generated when both the $K$-way and Matching classifiers agree with high confidence. This mutual agreement reduces the risk of confirmation bias. As the model improves, it identifies better descriptive words for each class, updating the CSR and further refining the Matching classifier.

Key Insights

  • BERT as a Matchmaker: BERT’s attention mechanism naturally aligns sentence tokens with class-related words. This zero-shot matching signal is a powerful classification cue.
  • Stability through Agreement: Requiring consensus between two distinct classification heads ensures that the model only learns from high-confidence, consistent pseudo-labels.
  • Dynamic Class Definitions: PCM’s CSR is not static. It evolves to include more nuanced keywords (e.g., "Iraq" or "Military" for "World News"), leading to sharper decision boundaries.
Visualization of how the model aligns sentence tokens with class-related keywords and calculates cosine similarity across categories.

Experimental Results

The authors evaluated PCM on four benchmarks: AG News, DBpedia, Yahoo! Answers, and IMDB, focusing on low-resource settings (3 to 50 labels per class).

  • State-of-the-Art Performance: PCM outperformed existing methods like UDA and MixText. Notably, on the IMDB dataset with only 3 labels per class, PCM achieved 73.86% accuracy, a significant jump from UDA's 63.01%.
  • Reduced Sensitivity: PCM showed lower standard error across different labeled sets, indicating higher stability regardless of which specific samples were labeled.
  • Ablation Success: The dynamic update of CSR proved critical. On the 3-label Yahoo! dataset, removing progressive updates caused accuracy to plummet from 63.52% to 39.49%.

Limitations and Future Work

While effective, PCM was primarily tested on encoder-only models like BERT. Future work could explore decoder-only (GPT) or permutation-based (XLNet) architectures. Additionally, the study was limited to English; cross-lingual applicability remains an open question. Finally, the dual-classifier architecture introduces additional computational overhead compared to standard fine-tuning.

Summary updated

Abstract

Semi-supervised learning is a promising way to reduce the annotation cost for text-classification. Combining with pre-trained language models (PLMs), e.g., BERT, recent semi-supervised learning methods achieved impressive performance. In this work, we further investigate the marriage between semi-supervised learning and a pre-trained language model. Unlike existing approaches that utilize PLMs only for model parameter initialization, we explore the inherent topic matching capability inside PLMs for building a more powerful semi-supervised learning approach. Specifically, we propose a joint semi-supervised learning process that can progressively build a standard $K$-way classifier and a matching network for the input text and the Class Semantic Representation (CSR). The CSR will be initialized from the given labeled sentences and progressively updated through the training process. By means of extensive experiments, we show that our method can not only bring remarkable improvement to baselines, but also overall be more stable, and achieves state-of-the-art performance in semi-supervised text classification.
BibTeX
@inproceedings{xu2022progressive,
  author = {Hai-Ming Xu and Lingqiao Liu and Ehsan Abbasnejad},
  title = {{Progressive Class Semantic Matching for Semi-supervised Text Classification}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.18653/v1/2022.naacl-main.219},
}
Figure