Abstract
Fine-tuning large language models on downstream tasks with limited labeled data often results in instability and poor performance. This paper proposes Contrastive Masked Language Modeling (CMLM), a post-training framework that utilizes a novel "Complementary Random Masking" (CRM) strategy. CMLM integrates token-level and sequence-level contrastive learning to align representations on unlabeled task data, significantly enhancing few-shot performance without requiring expensive external data augmentation.
Introduction
The standard paradigm in NLP involves taking a model pre-trained on massive general corpora (like RoBERTa) and fine-tuning it on task-specific labeled data. However, when labeled data is scarce—the few-shot scenario—this process becomes highly unstable and prone to overfitting.
A common mitigation strategy is post-training (or domain-adaptive pre-training), which continues unsupervised pre-training on unlabeled examples from the target task before supervised fine-tuning. Existing post-training methods usually focus on either token-level learning (e.g., standard Masked Language Modeling) or sequence-level contrastive learning (often requiring expensive augmentation like back-translation). CMLM addresses this gap by proposing a computationally efficient way to integrate both granularities during the post-training phase.
Methods
The CMLM framework relies on two primary innovations to bridge token-level and sequence-level learning:
- Complementary Random Masking (CRM): Instead of using complex data augmentations, the authors generate two masked versions of the same input sequence. Sequence A uses standard dynamic masking (15% probability). Sequence B uses a higher probability (70%) but is constrained: if a token was masked in Sequence A, it cannot be masked in Sequence B. This forces the model to reconstruct the sequence from entirely different contexts.
- Bi-Granularity Loss: The model applies standard Masked Language Modeling (MLM) to Sequence A for token-level similarity. Simultaneously, it applies a contrastive learning loss between the encoded representations of Sequence A and Sequence B to capture sequence-level similarities.
To compute the sequence-level loss efficiently, the authors adapt the SimSiam framework from computer vision. Unlike SimCLR, SimSiam does not require negative samples, making it significantly more memory-efficient for NLP tasks.
Experimental Setup
The model was evaluated on 8 classification tasks from the GLUE benchmark. To simulate few-shot environments, training subsets of 20, 100, and 1000 labeled examples were created. The remaining training data for each task was stripped of labels and used as the unsupervised corpus for the post-training phase.
Key Results and Insights
CMLM demonstrates superior performance in low-resource regimes, particularly as the amount of unlabeled task data increases.
- Few-Shot Superiority: In 100-shot and 1000-shot scenarios, CMLM outperformed baselines—including standard fine-tuning (FT), Supervised Contrastive Learning (SCL), and standard MLM post-training (TAPT)—in 13 out of 16 task settings.
- Ablation Studies: Removing the MLM loss dropped performance by up to 7.5%, while removing the contrastive learning (CL) loss caused a smaller drop (up to 1.4%). This suggests that while bi-granularity is optimal, MLM remains the foundational component of the representation learning.
- CRM Effectiveness: Swapping CRM for standard Dynamic Random Masking (DRM) resulted in a performance drop of up to 1.6% accuracy, validating the complementary approach.
Conclusion
CMLM provides a robust framework for improving few-shot NLP performance by aligning representations at both token and sequence levels. While the method shows significant gains at 100 and 1000 examples, it remains unstable in extreme few-shot settings (e.g., 20 examples), where variance across random seeds is high. Future work will explore the mechanical effects of bi-granularity on domain transfer and more advanced sequence-level contrastive techniques.
Summary updated