Protum: A New Method For Prompt Tuning Based on "[MASK]"

Pan He, Yuxi Chen, Yan Wang, Yanru Zhang

Abstract

Recently, prompt tuning has gradually become a new paradigm for NLP, which only depends on the representation of the words by freezing the parameters of pre-trained language models (PLMs) to obtain remarkable performance on downstream tasks. It maintains the consistency of the Masked Language Model (MLM) task in the process of pre-training and avoids some issues that may happen during fine-tuning. Naturally, we consider that the "[MASK]" tokens carry more useful information than other tokens because the model combines with context to predict the masked tokens. Among the current prompt tuning methods, there is a serious problem of random composition of the answer tokens in prediction when they predict multiple words, requiring them to map tokens to labels with the help of a verbalizer. In response to this issue, we propose a new Prompt Tuning based on "[MASK]" (Protum) method, which constructs a classification task through the information carried by the hidden layer of "[MASK]" tokens and then predicts the labels directly rather than the answer tokens. At the same time, we explore how different hidden layers under "[MASK]" impact our classification model on many different datasets. Finally, we find that our Protum can achieve much better performance than fine-tuning after continuous pre-training with less time consumption. Our model facilitates the practical application of large models in NLP.

Introduction

Fine-tuning large pre-trained language models (PLMs) like BERT or RoBERTa has become the standard for NLP, but it is computationally expensive, requiring the update and storage of billions of parameters for every new task. Prompt Tuning (PT) emerged as a lightweight alternative: by freezing the model and using a "template" (e.g., "[X]. Overall, this is a [MASK] movie."), classification tasks are transformed into fill-in-the-blank tasks.

However, standard PT faces two major challenges. First, the Verbalizer Problem: researchers must manually or automatically map predicted words (like "great" or "terrible") to labels (like "Positive" or "Negative"). This mapping is often arbitrary and becomes complex when labels require multiple tokens, particularly in languages like Chinese. Second, Information Underutilization: most PT methods only utilize the final layer of the model or a specific word prediction, ignoring the rich features embedded in the model's intermediate hidden layers.

Methodology

To address these limitations, we propose Protum, which rethinks the role of the [MASK] token in prompt tuning.

Protum-base

Instead of asking the model to predict a specific word at the [MASK] position and then using a verbalizer, Protum-base extracts the hidden state vector of the [MASK] token directly. This vector is fed into a simple, trainable linear classifier. This approach removes the need for a verbalizer entirely, as the model learns to map the internal representation of the mask directly to the class label.

Comparison between standard prompt tuning and the proposed Protum-base architecture.

Protum-Res

To further enhance performance, we introduced Protum-Res, an architecture inspired by residual networks. Standard prompt tuning often suffers from being "shallow." Protum-Res adds small residual blocks that bridge multiple hidden layers of the frozen PLM.

The Protum-Res architecture utilizing residual blocks to bridge multiple hidden layers of a frozen PLM.

By taking pooled hidden states from various layers and combining them using residual connections and ReLU activations, the model can leverage information from across the entire depth of the PLM without needing to unfreeze the PLM’s parameters. We also utilize Continuous Pre-training, training the model on specific prompt templates using standard MLM before the final tuning phase to align the model with the new prompt structure.

Experimental Setup

We evaluated Protum on three sentence-pair matching tasks from the SuperGLUE benchmark, representing different data scales:

  • CB (CommitmentBank): A small dataset with 250 training samples.
  • RTE (Recognizing Textual Entailment): A medium dataset with 2,500 samples.
  • BoolQ (Boolean Questions): A larger dataset with 9,427 samples.

Key Results and Discussion

Protum demonstrated significant gains over both standard Fine-Tuning (FT) and other Prompt Tuning (PT) methods, particularly on smaller datasets.

  • Superiority on Small Data: On the CB dataset, Protum-Res achieved 94.6% accuracy, significantly outperforming standard fine-tuning's 83.7%. The frozen nature of the PLM in Protum prevents the catastrophic forgetting or overfitting often seen in full fine-tuning on limited data.
  • Competitive Performance: On the RTE dataset, Protum-Res reached 76.9% accuracy, surpassing standard FT (71.6%).
  • The "Sweet Spot" Layer: Our analysis revealed that the third-to-last hidden layer (H[-3]) often contains more useful information for classification than the very last layer, suggesting that the final layers of PLMs may be overly specialized for the MLM task rather than general semantic representation.
  • Hyperparameter Sensitivity: The performance of Protum-Res is sensitive to the step size $K$ (number of layers a residual unit crosses) and the starting layer $S$. For BERT-large, we found that a $K$ value of 3 and specific starting layers yielded the highest accuracy.

Conclusion

Protum offers a more efficient and effective way to leverage large pre-trained models by focusing on the information-rich [MASK] hidden states and utilizing residual connections. While a small performance gap remains on larger datasets like BoolQ (75.8% for Protum-Res vs 77.4% for FT), the method's efficiency and robustness on small-to-medium data make it a powerful tool for practical NLP applications. Future work will explore the robustness of Protum across different "hard prompt" templates and further refine the residual unit architecture.

Summary updated

Abstract

Recently, prompt tuning \cite{lester2021power} has gradually become a new paradigm for NLP, which only depends on the representation of the words by freezing the parameters of pre-trained language models (PLMs) to obtain remarkable performance on downstream tasks. It maintains the consistency of Masked Language Model (MLM) \cite{devlin2018bert} task in the process of pre-training, and avoids some issues that may happened during fine-tuning. Naturally, we consider that the "[MASK]" tokens carry more useful information than other tokens because the model combines with context to predict the masked tokens. Among the current prompt tuning methods, there will be a serious problem of random composition of the answer tokens in prediction when they predict multiple words so that they have to map tokens to labels with the help verbalizer. In response to the above issue, we propose a new \textbf{Pro}mpt \textbf{Tu}ning based on "[\textbf{M}ASK]" (\textbf{Protum}) method in this paper, which constructs a classification task through the information carried by the hidden layer of "[MASK]" tokens and then predicts the labels directly rather than the answer tokens. At the same time, we explore how different hidden layers under "[MASK]" impact on our classification model on many different data sets. Finally, we find that our \textbf{Protum} can achieve much better performance than fine-tuning after continuous pre-training with less time consumption. Our model facilitates the practical application of large models in NLP.
BibTeX
@misc{he2022protum,
  author = {Pan He and Yuxi Chen and Yan Wang and Yanru Zhang},
  title = {{Protum: A New Method For Prompt Tuning Based on "[MASK]"}},
  year = {2022},
  eprint = {2201.12109},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2201.12109},
}
Figure