Making Pretrained Language Models Good Long-tailed Learners

Chen Zhang, Lei Ren, Jingang Wang, Wei Wu, Dawei Song

TL;DR

This paper demonstrates that Prompt-tuning significantly outperforms standard fine-tuning (CLS-tuning) in long-tailed classification tasks by effectively addressing the "tail bottleneck." The authors reveal a surprising insight: the success of Prompt-tuning isn't primarily due to the cloze-style input prompt, but rather its reuse of the pretrained Masked Language Modeling (MLM) head structure and parameterization.

What Problem Does This Paper Address?

In real-world classification tasks, data often follows a long-tailed distribution: a few "head" classes have abundant examples, while many "tail" classes have very few. Standard fine-tuning—where we attach a fresh classifier head to the [CLS] token—tends to collapse on these tail classes because they lack enough data to train new parameters from scratch. This is known as the tail bottleneck.

Figure 1: An example long-tailed class distribution from IFLYTEK dataset (Xu et al., 2020), where we can distinguish tail classes from head classes.

While Prompt-tuning has revolutionized few-shot learning, its potential for long-tailed scenarios (which are essentially collections of few-shot problems) hadn't been systematically explored. The authors set out to prove that Prompt-tuning makes Pretrained Language Models (PLMs) "Good Long-tailed Learners" (GLEE).

Methods

The researchers compared two primary schemes:

  1. CLS-tuning: The standard approach of adding a Tanh-activated MLP on top of the [CLS] representation.
  2. Prompt-tuning: Converting the task into a "cloze" test (e.g., "The movie was [MASK].") and using the pretrained MLM head to predict the missing word.
Figure 2: Illustration of two finetuning schemes.

To understand why Prompt-tuning works better, they conducted a "bridge" analysis, progressively modifying CLS-tuning to look more like Prompt-tuning. They tested the impact of:

  • Input Structure: Giving CLS-tuning a prompt-like input.
  • Classifier Structure: Replacing the Tanh MLP with the MLM head's structure (Linear -> GELU -> LayerNorm -> Linear).
  • Parameterization: Initializing the LayerNorm (LN) with pretrained weights from the MLM head.

Datasets

The authors evaluated their hypothesis on five diverse long-tailed datasets:

  • CMID: Chinese medical intent detection.
  • IFLYTEK: App category classification.
  • CTC: Clinical trial criterion categorization.
  • MSRA: Entity typing.
  • R52: Document topic classification (Reuters).

They also verified their findings on few-shot benchmarks like ECOM, RTE, and BOOLQ.

Key Insights and Ideas

  • The Classifier is King: Surprisingly, the "prompt" (the input structure) is less important than the "head" (the classifier structure). Reusing the MLM head's architecture and its pretrained LayerNorm (LN) parameters is what actually drives the performance gains.
  • The Decoupling Property: A good long-tailed learner should have classification weight norms that are relatively uniform across classes. CLS-tuning fails here, showing much larger weights for head classes. Prompt-tuning maintains a much flatter distribution, preventing the model from being biased toward the head.
Figure 3: Visualization of weight norms. Prompt-tuning (red) maintains a much more stable weight distribution across class indices compared to CLS-tuning (blue).
  • Solving the "Dying ReLU" Problem: In tail classes with few examples, activations can easily drop to zero (the dying ReLU problem), leading to information loss. The authors found that the pretrained LayerNorm from the MLM head helps "re-activate" these features and re-scale them based on knowledge learned during pretraining.
Figure 4: An illustration of sampled feature distribu- tions, for both the head and tail.

Key Results

  • Significant F1 Gains: Across all datasets, Prompt-tuning outperformed CLS-tuning by large margins. On average, the Macro F1 score jumped from 62.6% to 70.4%.
  • Tail Performance: The improvement was most dramatic on tail classes. For example, on the IFLYTEK dataset, tail class F1 improved from 27.4% to 40.1%.
  • Improved CLS-tuning: By simply equipping standard CLS-tuning with a ReLU activation and pretrained LayerNorm (dubbed "CLS-tuning w/ pt. LN"), the authors achieved performance nearly identical to Prompt-tuning (68.7% avg F1), proving that the prompt itself isn't the only way to unlock PLM potential.

Limitations and Future Work

  • Template Sensitivity: The authors acknowledge that Prompt-tuning is sensitive to the choice of templates, though they did not focus on finding the "optimal" template for this study.
  • Scalability: While they tested on BERT-base models, future work could explore if these findings hold for much larger generative models (like GPT-3/4) or different architectures like T5.
  • Beyond Classification: The study focuses on classification; extending these insights to other long-tailed NLP tasks like relation extraction or open-domain QA remains an open area.

Summary updated

Abstract

Prompt-tuning has shown appealing performance in few-shot classification by virtue of its capability in effectively exploiting pre-trained knowledge. This motivates us to check the hypothesis that prompt-tuning is also a promising choice for long-tailed classification, since the tail classes are intuitively few-shot ones. To achieve this aim, we conduct empirical studies to examine the hypothesis. The results demonstrate that prompt-tuning makes pretrained language models at least good long-tailed learners. For intuitions on why prompt-tuning can achieve good performance in long-tailed classification, we carry out in-depth analyses by progressively bridging the gap between prompt-tuning and commonly used finetuning. The summary is that the classifier structure and parameterization form the key to making good long-tailed learners, in comparison with the less important input structure. Finally, we verify the applicability of our finding to few-shot classification. Good long-tailed learners can be abbreviated as Glee.
BibTeX
@inproceedings{zhang2022making,
  author = {Chen Zhang and Lei Ren and Jingang Wang and Wei Wu and Dawei Song},
  title = {{Making Pretrained Language Models Good Long-tailed Learners}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=MH5BazWAeE},
}
Figure