Progressive Class Semantic Matching for Semi-supervised Text Classification

Hai-Ming Xu, Lingqiao Liu, Ehsan Abbasnejad

Abstract

Semi-supervised learning is a promising way to reduce the annotation cost for text-classification. Combining with pre-trained language models (PLMs), e.g., BERT, recent semi-supervised learning methods achieved impressive performance. In this work, we further investigate the marriage between semi-supervised learning and a pre-trained language model. Unlike existing approaches that utilize PLMs only for model parameter initialization, we explore the inherent topic matching capability inside PLMs for building a more powerful semi-supervised learning approach. Specifically, we propose a joint semi-supervised learning process that can progressively build a standard $K$-way classifier and a matching network for the input text and the Class Semantic Representation (CSR). The CSR will be initialized from the given labeled sentences and progressively updated through the training process. By means of extensive experiments, we show that our method can not only bring remarkable improvement to baselines, but also overall be more stable, and achieves state-of-the-art performance in semi-supervised text classification.

Introduction

Text classification is a fundamental task in Natural Language Processing (NLP), yet training high-performing models typically requires extensive labeled datasets. Semi-supervised learning (SSL) addresses this bottleneck by leveraging small amounts of labeled data alongside large volumes of unlabeled text. While recent SSL methods utilize Pre-trained Language Models (PLMs) like BERT as robust feature extractors, they often overlook the inherent semantic matching capabilities embedded within these models during pre-training. This paper introduces Progressive Class Semantic Matching (PCM), a framework designed to tap into BERT's internal "matchmaker" to improve classification accuracy and stability, particularly in low-resource settings.

The PCM Framework

The core innovation of PCM is the joint optimization of a standard classifier and a semantic matching network. The architecture consists of three primary components:

  1. The $K$-way Classifier: A standard classification head that predicts the category of an input text based on global features.
  2. Class Semantic Representation (CSR): Each class is represented by a set of semantically related words. For example, a "Sports" class might be represented by words like football, stadium, and score. These representations are initialized from labeled data and updated progressively.
  3. The Matching Classifier: This component evaluates the semantic alignment between input text and the CSR of specific classes.
The PCM architecture integrates a standard K-way classifier with a matching classifier that leverages Class Semantic Representations (CSR).

Joint Training and Pseudo-Labeling

PCM enhances the Unsupervised Data Augmentation (UDA) process. When processing unlabeled data, the model generates a pseudo-label only when the $K$-way classifier and the Matching classifier both exhibit high confidence and, crucially, agree on the predicted label. This dual-agreement mechanism significantly reduces the noise typically associated with self-training in SSL.

Experimental Evaluation

The researchers evaluated PCM across four major benchmarks: AG News, DBpedia, Yahoo! Answers, and IMDB. The experiments focused on extreme low-resource scenarios, utilizing as few as 3 to 50 labeled examples per class.

Key Results

  • State-of-the-Art Performance: PCM consistently outperformed strong baselines such as UDA and MixText. The gains were most pronounced in the most data-scarce environments. For instance, on the IMDB dataset with only 3 labels per class, PCM achieved 73.86% accuracy, a substantial improvement over MixText's 56.27%.
  • Enhanced Stability: SSL methods are often prone to high variance depending on the specific labeled samples chosen. PCM demonstrated significantly lower standard errors, indicating a more reliable training process.
  • Dynamic Semantics: The progressive updating of the CSR allows the model to adapt its understanding of class definitions to the specific vocabulary of the target dataset over time.

Analysis of Semantic Matching

BERT’s pre-training (specifically Next Sentence Prediction) equips it with an inherent ability to relate sentences to topic-specific keywords. PCM formalizes this by appending CSR words to input sequences. As shown in the visualizations, the model's internal attention and similarity scores shift dynamically when relevant semantic anchors are provided.

Visualization of how appending different semantic words (e.g., 'boyfriend' for family, 'football' for sports) influences the cosine similarity and model attention across different classes.

Conclusion

Progressive Class Semantic Matching (PCM) represents a significant step forward in semi-supervised text classification. By moving beyond using BERT as a static encoder and instead leveraging its internal matching logic, PCM achieves superior accuracy and stability. Future work may involve extending this framework to auto-regressive models like GPT and exploring its efficacy across diverse languages beyond English.

Summary updated

Abstract

Semi-supervised learning is a promising way to reduce the annotation cost for text-classification. Combining with pre-trained language models (PLMs), e.g., BERT, recent semi-supervised learning methods achieved impressive performance. In this work, we further investigate the marriage between semi-supervised learning and a pre-trained language model. Unlike existing approaches that utilize PLMs only for model parameter initialization, we explore the inherent topic matching capability inside PLMs for building a more powerful semi-supervised learning approach. Specifically, we propose a joint semi-supervised learning process that can progressively build a standard $K$-way classifier and a matching network for the input text and the Class Semantic Representation (CSR). The CSR will be initialized from the given labeled sentences and progressively updated through the training process. By means of extensive experiments, we show that our method can not only bring remarkable improvement to baselines, but also overall be more stable, and achieves state-of-the-art performance in semi-supervised text classification.
BibTeX
@inproceedings{xu2022progressive,
  author = {Hai-Ming Xu and Lingqiao Liu and Ehsan Abbasnejad},
  title = {{Progressive Class Semantic Matching for Semi-supervised Text Classification}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.18653/v1/2022.naacl-main.219},
}
Figure