Abstract
Deep learning models for text classification typically require extensive labeled datasets. Semi-supervised learning (SSL) attempts to mitigate this by utilizing unlabeled data. However, many SSL approaches fail to fully exploit the internal semantic knowledge of pre-trained language models (PLMs) like BERT. This paper introduces Progressive Class Semantic Matching (PCM), a framework that treats BERT as more than a feature extractor. PCM utilizes BERT’s inherent ability to match sentences with topic-related words through a dual-classifier system and dynamic class semantic representations. Experimental results show that PCM achieves state-of-the-art performance in extreme low-resource scenarios, such as having only three labeled examples per class.
Introduction
The primary challenge in semi-supervised text classification is effectively leveraging a small amount of labeled data alongside a large pool of unlabeled text. While fine-tuning PLMs has become standard, the authors argue that standard fine-tuning ignores the rich internal knowledge BERT acquires during pre-training tasks like Next Sentence Prediction. BERT is naturally adept at recognizing semantic relationships between sentences and specific "class-related" words. PCM formalizes this matching capability to create a more robust SSL framework.
Methods: The PCM Framework
The PCM model integrates three co-evolving components to improve classification accuracy progressively:
- Standard $K$-way Classifier: A traditional layer predicting one of $K$ categories.
- Class Semantic Representation (CSR): Instead of abstract indices, classes are represented as average embeddings of related words (e.g., "football" for "Sports").
- Matching Classifier: This component calculates the semantic "match" between an input sentence and the CSR of a class.
The training process employs a "Progressive" mechanism. Initially, keywords are identified from the small labeled set. During training on unlabeled data (using Unsupervised Data Augmentation), pseudo-labels are only generated when both the $K$-way and Matching classifiers agree with high confidence. This mutual agreement reduces the risk of confirmation bias. As the model improves, it identifies better descriptive words for each class, updating the CSR and further refining the Matching classifier.
Key Insights
- BERT as a Matchmaker: BERT’s attention mechanism naturally aligns sentence tokens with class-related words. This zero-shot matching signal is a powerful classification cue.
- Stability through Agreement: Requiring consensus between two distinct classification heads ensures that the model only learns from high-confidence, consistent pseudo-labels.
- Dynamic Class Definitions: PCM’s CSR is not static. It evolves to include more nuanced keywords (e.g., "Iraq" or "Military" for "World News"), leading to sharper decision boundaries.
Experimental Results
The authors evaluated PCM on four benchmarks: AG News, DBpedia, Yahoo! Answers, and IMDB, focusing on low-resource settings (3 to 50 labels per class).
- State-of-the-Art Performance: PCM outperformed existing methods like UDA and MixText. Notably, on the IMDB dataset with only 3 labels per class, PCM achieved 73.86% accuracy, a significant jump from UDA's 63.01%.
- Reduced Sensitivity: PCM showed lower standard error across different labeled sets, indicating higher stability regardless of which specific samples were labeled.
- Ablation Success: The dynamic update of CSR proved critical. On the 3-label Yahoo! dataset, removing progressive updates caused accuracy to plummet from 63.52% to 39.49%.
Limitations and Future Work
While effective, PCM was primarily tested on encoder-only models like BERT. Future work could explore decoder-only (GPT) or permutation-based (XLNet) architectures. Additionally, the study was limited to English; cross-lingual applicability remains an open question. Finally, the dual-classifier architecture introduces additional computational overhead compared to standard fine-tuning.
Summary updated