Abstract
Prompt-based learning has emerged as a powerful paradigm for few-shot text classification, yet it remains vulnerable to misleading lexical cues within input text. We present Multi-Mask Label Mapping (MMLM), a novel technique that enhances model robustness by leveraging an ensemble of mask slots. By extracting multiple keywords and generating augmented prompts, MMLM effectively dilutes the influence of distracting tokens. Our experiments across five benchmark datasets demonstrate that MMLM achieves state-of-the-art performance in few-shot scenarios without relying on external knowledge bases.
Introduction
The transition from traditional fine-tuning to prompt-based learning has significantly improved the performance of Large Language Models (LLMs) like BERT and RoBERTa on low-resource tasks. In this "cloze-style" approach, classification is reframed as a fill-in-the-blank task. However, LLMs are often overly sensitive to specific lexical cues—individual words that strongly bias the model toward a particular label, regardless of the broader context. For instance, a sentiment analysis model might incorrectly classify the sentence "Boring starting but overall ok and worth watching" as negative due to the presence of the word "boring." Standard single-mask prompts are highly susceptible to such local distractions.
Methodology
To address this sensitivity, we propose the Multi-Mask Label Mapping (MMLM) framework, which shifts the focus from a single lexical cue to a broader "forest" of keywords.
Cloze-Based Augmentation
The first stage of MMLM involves identifying the top-$n$ keywords ($k_1, k_2, \dots, k_n$) that the model perceives as most relevant to the input text $x$. For each keyword, an augmented prompt is generated using a stimulation template (e.g., "Keyword: {keyword}. It was [MASK]."). This process creates $n$ different "views" of the same input, each anchored by a different lexical cue.
Multi-Mask Scheme
Each augmented input is embedded and processed through the model $M_\theta$. To ensure diversity among the predictions and prevent the ensemble from collapsing into a single biased state, we introduce controlled noise via embedding perturbation and dropout. The final classification is determined by aggregating the predictions from all $n$ masks using either weighted voting (based on keyword importance $w_i$) or plurality voting.
Experimental Setup
We evaluated MMLM on five diverse benchmarks: AG’s News (topic), IMDB (sentiment), Amazon (sentiment), DBPedia (topic), and Yahoo (topic). Testing was conducted in $K$-shot settings ($K \in \{1, 5, 10, 20\}$) to assess performance under extreme data scarcity.
Results and Discussion
MMLM consistently outperformed existing label mapping techniques, including manual, search-based, and soft-prompt methods. Notably, it surpassed Knowledgeable Prompt-tuning (KPT), the previous state-of-the-art, which requires external knowledge bases.
Key Findings
- Robustness to Noise: By aggregating multiple mask predictions, the impact of a single misleading keyword is mathematically diluted. Theoretical analysis using Hoeffding’s inequality confirms that the error probability decreases as the number of masks $n$ increases.
- Internal Knowledge Exploitation: MMLM proves that the latent knowledge within pre-trained LLMs is sufficient for high-accuracy mapping if properly stimulated, removing the need for complex external knowledge integration.
- Few-Shot Excellence: On the IMDB dataset with $K=1$, MMLM achieved 92.6% accuracy, a significant improvement over the 50.8% achieved by traditional fine-tuning.
Conclusion
Multi-Mask Label Mapping provides a robust solution to the lexical sensitivity of prompt-based models. By treating classification as an ensemble task over multiple internal keywords, MMLM achieves superior few-shot performance. While the method introduces additional computational overhead due to multiple forward passes, the gains in accuracy and the elimination of external dependencies make it a compelling choice for low-resource NLP tasks.
Summary updated