Multi-Mask Label Mapping for Prompt-Based Learning

Jirui Qi, Richong Zhang, Jaein Kim, Junfan Chen, Wenyi Qin, Yongyi Mao

Abstract

Prompt-based learning has emerged as a powerful paradigm for few-shot text classification, yet it remains vulnerable to misleading lexical cues within input text. We present Multi-Mask Label Mapping (MMLM), a novel technique that enhances model robustness by leveraging an ensemble of mask slots. By extracting multiple keywords and generating augmented prompts, MMLM effectively dilutes the influence of distracting tokens. Our experiments across five benchmark datasets demonstrate that MMLM achieves state-of-the-art performance in few-shot scenarios without relying on external knowledge bases.

Introduction

The transition from traditional fine-tuning to prompt-based learning has significantly improved the performance of Large Language Models (LLMs) like BERT and RoBERTa on low-resource tasks. In this "cloze-style" approach, classification is reframed as a fill-in-the-blank task. However, LLMs are often overly sensitive to specific lexical cues—individual words that strongly bias the model toward a particular label, regardless of the broader context. For instance, a sentiment analysis model might incorrectly classify the sentence "Boring starting but overall ok and worth watching" as negative due to the presence of the word "boring." Standard single-mask prompts are highly susceptible to such local distractions.

Methodology

To address this sensitivity, we propose the Multi-Mask Label Mapping (MMLM) framework, which shifts the focus from a single lexical cue to a broader "forest" of keywords.

Cloze-Based Augmentation

The first stage of MMLM involves identifying the top-$n$ keywords ($k_1, k_2, \dots, k_n$) that the model perceives as most relevant to the input text $x$. For each keyword, an augmented prompt is generated using a stimulation template (e.g., "Keyword: {keyword}. It was [MASK]."). This process creates $n$ different "views" of the same input, each anchored by a different lexical cue.

Multi-Mask Scheme

Each augmented input is embedded and processed through the model $M_\theta$. To ensure diversity among the predictions and prevent the ensemble from collapsing into a single biased state, we introduce controlled noise via embedding perturbation and dropout. The final classification is determined by aggregating the predictions from all $n$ masks using either weighted voting (based on keyword importance $w_i$) or plurality voting.

The MMLM framework architecture, illustrating the transition from keyword extraction to multi-mask prediction.

Experimental Setup

We evaluated MMLM on five diverse benchmarks: AG’s News (topic), IMDB (sentiment), Amazon (sentiment), DBPedia (topic), and Yahoo (topic). Testing was conducted in $K$-shot settings ($K \in \{1, 5, 10, 20\}$) to assess performance under extreme data scarcity.

Results and Discussion

MMLM consistently outperformed existing label mapping techniques, including manual, search-based, and soft-prompt methods. Notably, it surpassed Knowledgeable Prompt-tuning (KPT), the previous state-of-the-art, which requires external knowledge bases.

Performance comparison of MMLM (Weighted and Plurality) against baseline and SOTA methods across five datasets and various K-shot settings.

Key Findings

  • Robustness to Noise: By aggregating multiple mask predictions, the impact of a single misleading keyword is mathematically diluted. Theoretical analysis using Hoeffding’s inequality confirms that the error probability decreases as the number of masks $n$ increases.
  • Internal Knowledge Exploitation: MMLM proves that the latent knowledge within pre-trained LLMs is sufficient for high-accuracy mapping if properly stimulated, removing the need for complex external knowledge integration.
  • Few-Shot Excellence: On the IMDB dataset with $K=1$, MMLM achieved 92.6% accuracy, a significant improvement over the 50.8% achieved by traditional fine-tuning.

Conclusion

Multi-Mask Label Mapping provides a robust solution to the lexical sensitivity of prompt-based models. By treating classification as an ensemble task over multiple internal keywords, MMLM achieves superior few-shot performance. While the method introduces additional computational overhead due to multiple forward passes, the gains in accuracy and the elimination of external dependencies make it a compelling choice for low-resource NLP tasks.

Summary updated

Abstract

Prompt-based Learning has shown significant success in few-shot classification. The mainstream approach is to concatenate a template for the input text to transform the classification task into a cloze-type task where label mapping plays an important role in finding the ground-truth labels. While current label mapping methods only use the contexts in one single input, it could be crucial if wrong information is contained in the text. Specifically, it is proved in recent work that even the large language models like BERT/RoBERTa make classification decisions heavily dependent on a specific keyword regardless of the task or the context. Such a word is referred to as a lexical cue and if a misleading lexical cue is included in the instance it will lead the model to make a wrong prediction. We propose a multi-mask prompt-based approach with Multi-Mask Label Mapping (MMLM) to reduce the impact of misleading lexical cues by allowing the model to exploit multiple lexical cues. To satisfy the conditions of few-shot learning, an instance augmentation approach for the cloze-type model is proposed and the misleading cues are gradually excluded through training. We demonstrate the effectiveness of MMLM by both theoretical analysis and empirical studies, and show that MMLM outperforms other existing label mapping approaches.
BibTeX
@inproceedings{qi2023multi,
  author = {Jirui Qi and Richong Zhang and Jaein Kim and Junfan Chen and Wenyi Qin and Yongyi Mao},
  title = {{Multi-Mask Label Mapping for Prompt-Based Learning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i11.26579},
}
Figure