Abstract
Deep learning models for text classification often require vast amounts of labeled data, which is frequently unavailable in specialized domains. This paper introduces CformerM, a semi-supervised framework that enhances the teacher-student "Meta Pseudo Labels" architecture. The primary innovation is Objective Masking, a pre-training strategy that uses Latent Dirichlet Allocation (LDA) to identify and mask topically significant words. By forcing the model to reconstruct domain-relevant terms rather than random tokens, CformerM achieves state-of-the-art performance in few-shot scenarios across multiple languages and technical domains.
Introduction
Text classification is a fundamental task in NLP, yet the reliance on "gold-labeled" data remains a significant bottleneck. In niche areas like medical research or specific news categories, obtaining expert labels is costly and time-consuming. Semi-supervised learning (SSL) addresses this by leveraging large unlabeled corpora alongside minimal labeled data.
Standard pre-training for models like BERT typically employs "Random Masking," where tokens are hidden with equal probability. However, for classification tasks, not all words are equally informative. Masking common stop words or contextually neutral terms may be less efficient than focusing on words that define specific topics (e.g., "dividend" for business or "touchdown" for sports).
Methodology: CformerM and Objective Masking
The proposed CformerM model builds upon the Cformer architecture, which utilizes a teacher-student meta-learning loop. In this setup, a teacher model generates pseudo-labels for unlabeled data, and a student model learns from these labels. The teacher is then updated based on the student's performance on a small set of gold-labeled data.
The core contribution is the integration of Objective Masking into the pre-training phase. This process involves four distinct stages:
- Unsupervised Topic Discovery: Latent Dirichlet Allocation (LDA) is applied to the entire dataset to identify $K$ latent topics.
- Word Selection: A relevance measure identifies "indicative" words—those that appear frequently within a specific topic but rarely elsewhere.
- Objective Masking: The Transformer encoders are pre-trained by specifically masking these topic-relevant words, forcing the model to become "topically sensitive."
- Meta Pseudo Labeling: The pre-trained encoders are integrated into the Cformer teacher-student training loop.
Experimental Setup
The researchers evaluated CformerM across diverse datasets and languages:
- Yahoo! Answers (English): 10 classes of general Q&A.
- AG News (English): 4 classes of short news snippets.
- Medical Abstracts (English): 5 classes of technical medical text.
- Bonnier News (Swedish): A real-world, highly imbalanced dataset with 17 news categories.
Key Results and Insights
CformerM consistently outperformed state-of-the-art baselines, including standard Cformer, MixText, and UDA.
Few-Shot Performance
The benefits of Objective Masking are most pronounced when labeled data is extremely scarce. In a 10-shot setting (10 examples per class), CformerM achieved an absolute accuracy increase of 2.7% on Yahoo! Answers and 2.6% on AG News compared to standard BERT-based models.
Contextual Sensitivity
Qualitative analysis revealed that Objective Masking encourages the model to rely on broader context. For instance, in a sentence containing the word "steel" within a sports context, CformerM correctly classified the topic as "Sports," whereas random-masking models were often misled by the keyword "steel" into predicting "Business."
Domain Adaptation
The method proved particularly effective for specialized domains like medicine. Since medical terminology differs significantly from general-purpose pre-training data, the LDA-guided masking allowed the model to rapidly adapt to new, technical vocabularies.
Discussion and Limitations
While CformerM shows significant promise, certain challenges remain:
- Class Imbalance: On the Swedish Bonnier News dataset, extreme imbalance (e.g., 30,000 vs. 70 examples) hindered the LDA model's ability to find coherent topics for minority classes, limiting masking effectiveness.
- Text Length: Shorter texts, such as those in AG News, provide less context for the model to learn from during the pre-training phase compared to longer documents.
Future work will explore more sophisticated topic modeling techniques beyond LDA to better handle imbalanced data and short-form text, further refining the word selection process for objective masking.
Summary updated