Introduction
Fine-tuning large-scale pre-trained models like BERT or RoBERTa has become the standard for NLP tasks. However, in "few-shot" scenarios—where only a limited number of labeled examples are available—this process often suffers from extreme instability and poor generalization. To address this, researchers have turned to "post-training," an intermediate phase that adapts the model to the target domain using unlabeled data before final fine-tuning.
The CMLM Framework
The proposed method, Contrastive Masked Language Modeling (CMLM), introduces a bi-granularity approach to post-training. Unlike previous methods that focus solely on word-level patterns or sentence-level representations, CMLM bridges these two granularities without requiring computationally expensive data augmentation like back-translation.
Complementary Random Masking (CRM)
At the heart of CMLM is the Complementary Random Masking (CRM) strategy. Standard masking (Dynamic Random Masking) might hide the same tokens in different versions of a sentence, making contrastive tasks trivial. CRM ensures that for a given sentence, the model generates pairs of masked versions where the masked tokens are largely non-overlapping. This forces the model to rely on different semantic cues to recognize that both versions represent the same underlying sequence.
Bi-Granularity Learning
CMLM optimizes for two objectives simultaneously:
- Token-Level Similarity: The model performs standard Masked Language Modeling (MLM) by predicting hidden tokens based on their context.
- Sequence-Level Similarity: The model uses Contrastive Learning (CL) to ensure that different masked versions of the same sentence have similar representations in the vector space.
This dual focus allows the model to learn how specific words contribute to local context while maintaining a stable understanding of the overall sentence meaning, even when significant portions are missing.
Experimental Evaluation
The researchers evaluated CMLM on the GLUE benchmark, covering tasks such as sentiment analysis (SST-2), grammar (CoLA), and natural language inference (MNLI, QNLI, and RTE). To test few-shot performance, they created subsets with 20, 100, and 1,000 training examples.
Key Results
- Performance: CMLM outperformed strong baselines like Task-Adaptive Pre-Training (TAPT) in 13 out of 16 task configurations for the 100 and 1,000-shot settings.
- Efficiency: By using masking as a "natural" data augmenter, CMLM avoids the high computational costs associated with back-translation used in other contrastive methods.
- Stability: The method showed more consistent performance gains as the volume of unlabeled post-training data increased, although the extreme 20-shot scenario remains a challenge for all models due to high variance.
Conclusion
Bi-granularity contrastive learning provides a robust and efficient path for improving few-shot NLP. By leveraging the complementary nature of masking, CMLM captures both fine-grained word relationships and coarse-grained sequence semantics. Future work will explore the application of this method to broader domain transfer tasks and more complex sequence-level similarity measures.
Summary updated