TL;DR
This paper demonstrates that Prompt-tuning significantly outperforms standard fine-tuning (CLS-tuning) in long-tailed classification tasks by effectively addressing the "tail bottleneck." The authors reveal a surprising insight: the success of Prompt-tuning isn't primarily due to the cloze-style input prompt, but rather its reuse of the pretrained Masked Language Modeling (MLM) head structure and parameterization.
What Problem Does This Paper Address?
In real-world classification tasks, data often follows a long-tailed distribution: a few "head" classes have abundant examples, while many "tail" classes have very few. Standard fine-tuning—where we attach a fresh classifier head to the [CLS] token—tends to collapse on these tail classes because they lack enough data to train new parameters from scratch. This is known as the tail bottleneck.
While Prompt-tuning has revolutionized few-shot learning, its potential for long-tailed scenarios (which are essentially collections of few-shot problems) hadn't been systematically explored. The authors set out to prove that Prompt-tuning makes Pretrained Language Models (PLMs) "Good Long-tailed Learners" (GLEE).
Methods
The researchers compared two primary schemes:
- CLS-tuning: The standard approach of adding a Tanh-activated MLP on top of the
[CLS]representation. - Prompt-tuning: Converting the task into a "cloze" test (e.g., "The movie was [MASK].") and using the pretrained MLM head to predict the missing word.
To understand why Prompt-tuning works better, they conducted a "bridge" analysis, progressively modifying CLS-tuning to look more like Prompt-tuning. They tested the impact of:
- Input Structure: Giving CLS-tuning a prompt-like input.
- Classifier Structure: Replacing the Tanh MLP with the MLM head's structure (Linear -> GELU -> LayerNorm -> Linear).
- Parameterization: Initializing the LayerNorm (LN) with pretrained weights from the MLM head.
Datasets
The authors evaluated their hypothesis on five diverse long-tailed datasets:
- CMID: Chinese medical intent detection.
- IFLYTEK: App category classification.
- CTC: Clinical trial criterion categorization.
- MSRA: Entity typing.
- R52: Document topic classification (Reuters).
They also verified their findings on few-shot benchmarks like ECOM, RTE, and BOOLQ.
Key Insights and Ideas
- The Classifier is King: Surprisingly, the "prompt" (the input structure) is less important than the "head" (the classifier structure). Reusing the MLM head's architecture and its pretrained LayerNorm (LN) parameters is what actually drives the performance gains.
- The Decoupling Property: A good long-tailed learner should have classification weight norms that are relatively uniform across classes. CLS-tuning fails here, showing much larger weights for head classes. Prompt-tuning maintains a much flatter distribution, preventing the model from being biased toward the head.
- Solving the "Dying ReLU" Problem: In tail classes with few examples, activations can easily drop to zero (the dying ReLU problem), leading to information loss. The authors found that the pretrained LayerNorm from the MLM head helps "re-activate" these features and re-scale them based on knowledge learned during pretraining.
Key Results
- Significant F1 Gains: Across all datasets, Prompt-tuning outperformed CLS-tuning by large margins. On average, the Macro F1 score jumped from 62.6% to 70.4%.
- Tail Performance: The improvement was most dramatic on tail classes. For example, on the IFLYTEK dataset, tail class F1 improved from 27.4% to 40.1%.
- Improved CLS-tuning: By simply equipping standard CLS-tuning with a ReLU activation and pretrained LayerNorm (dubbed "CLS-tuning w/ pt. LN"), the authors achieved performance nearly identical to Prompt-tuning (68.7% avg F1), proving that the prompt itself isn't the only way to unlock PLM potential.
Limitations and Future Work
- Template Sensitivity: The authors acknowledge that Prompt-tuning is sensitive to the choice of templates, though they did not focus on finding the "optimal" template for this study.
- Scalability: While they tested on BERT-base models, future work could explore if these findings hold for much larger generative models (like GPT-3/4) or different architectures like T5.
- Beyond Classification: The study focuses on classification; extending these insights to other long-tailed NLP tasks like relation extraction or open-domain QA remains an open area.
Summary updated