Making Pretrained Language Models Good Long-tailed Learners

Chen Zhang, Lei Ren, Jingang Wang, Wei Wu, Dawei Song

Abstract

Prompt-tuning has shown appealing performance in few-shot classification by virtue of its capability in effectively exploiting pre-trained knowledge. This motivates us to check the hypothesis that prompt-tuning is also a promising choice for long-tailed classification, since the tail classes are intuitively few-shot ones. To achieve this aim, we conduct empirical studies to examine the hypothesis. The results demonstrate that prompt-tuning makes pretrained language models at least good long-tailed learners. For intuitions on why prompt-tuning can achieve good performance in long-tailed classification, we carry out in-depth analyses by progressively bridging the gap between prompt-tuning and commonly used finetuning. The summary is that the classifier structure and parameterization form the key to making good long-tailed learners, in comparison with the less important input structure. Finally, we verify the applicability of our finding to few-shot classification.

Introduction: The Long-Tail Challenge

In real-world scenarios, data is rarely distributed evenly. Most classification datasets exhibit a long-tailed distribution: a small number of "head" classes possess thousands of examples, while a vast majority of "tail" classes contain only a handful.

A typical long-tailed distribution where a few head classes dominate the dataset, leaving many tail classes with very few examples.

Standard Pretrained Language Models (PLMs) typically utilize CLS-tuning—adding a fresh linear layer on top of the [CLS] token—for classification tasks. However, CLS-tuning struggles significantly in long-tailed scenarios. The model becomes heavily biased toward head classes, and the tail classes suffer from a "tail bottleneck" where they are never properly optimized. While prompt-tuning (formulating the task as a "fill-in-the-blank" problem) has revolutionized few-shot learning, this research explores its potential to solve the broader long-tailed problem and investigates the underlying mechanisms of its success.

Methodology: Deconstructing Prompt-Tuning

The authors conduct a systematic comparison between standard CLS-tuning and prompt-tuning. To understand the "magic" behind prompt-tuning's success, they decompose the method into three distinct components:

  1. Shared Embeddings: Investigating whether sharing parameters between the classifier and the backbone improves performance.
  2. Input Structure: Determining if the "cloze-style" input (e.g., "It was [MASK].") is the primary driver of success.
  3. Classifier Structure: Analyzing the impact of reusing the pretrained Masked Language Model (MLM) head, which includes specific Layer Normalization and activation functions.

Through rigorous ablation studies, the researchers discovered that the classifier structure and parameterization are the most critical factors. They even proposed an "improved CLS-tuning" method that equips a standard classifier with ReLU activation and pretrained Layer Normalization (LN) from the MLM head. This hybrid approach manages to replicate much of prompt-tuning's gains without requiring a complex template.

Key Insights: Why It Works

The Weight Norm Distribution

In standard CLS-tuning, the weight norms for head classes are massive, while tail class norms are tiny. This disparity "tilts" the model's preference toward frequent classes. In contrast, prompt-tuning produces a much flatter weight distribution, meaning the model treats tail classes with nearly the same "importance" as head classes.

It’s the Head, Not the Prompt

Surprisingly, the "cloze-style" input structure is less important than the classifier head itself. Reusing the pretrained MLM head (Linear + LN + GELU/ReLU) allows the model to inherit knowledge about feature scaling. Standard classifiers often suffer from "dying neurons" where tail class features are zeroed out. The authors found that Layer Normalization, especially when initialized with pretrained weights, re-activates these dead features and facilitates knowledge transfer from head classes to tail classes.

Experimental Results

The hypothesis was evaluated on five diverse long-tailed classification datasets: CMID (medical intent), IFLYTEK (app categories), CTC (clinical trials), MSRA (entity typing), and R52 (document topics).

  • Superior Performance: Prompt-tuning outperformed CLS-tuning by an average of 7.8% in Macro F1 across all long-tailed datasets.
  • Dramatic Tail Gains: The improvement was most pronounced on tail classes. For example, on the IFLYTEK dataset, prompt-tuning improved tail class F1 from 27.4% to 40.1%.
  • Stability: Prompt-tuning demonstrated significantly lower variance across different runs compared to standard fine-tuning, indicating a more stable learning process.
  • Generalization: The "improved CLS-tuning" also showed strong results in few-shot settings (ECOM, RTE, and BOOLQ), proving that these architectural insights are broadly applicable beyond just long-tailed data.

Conclusion

This research demonstrates that prompt-tuning makes PLMs excellent long-tailed learners. The core of this success lies not in the prompt itself, but in the reuse of the pretrained MLM head and its associated normalization layers. By balancing weight distributions and preventing feature collapse in rare classes, this approach provides a robust framework for handling the data imbalances inherent in real-world applications.

Summary updated

Abstract

Prompt-tuning has shown appealing performance in few-shot classification by virtue of its capability in effectively exploiting pre-trained knowledge. This motivates us to check the hypothesis that prompt-tuning is also a promising choice for long-tailed classification, since the tail classes are intuitively few-shot ones. To achieve this aim, we conduct empirical studies to examine the hypothesis. The results demonstrate that prompt-tuning makes pretrained language models at least good long-tailed learners. For intuitions on why prompt-tuning can achieve good performance in long-tailed classification, we carry out in-depth analyses by progressively bridging the gap between prompt-tuning and commonly used finetuning. The summary is that the classifier structure and parameterization form the key to making good long-tailed learners, in comparison with the less important input structure. Finally, we verify the applicability of our finding to few-shot classification.
BibTeX
@inproceedings{zhang2022making,
  author = {Chen Zhang and Lei Ren and Jingang Wang and Wei Wu and Dawei Song},
  title = {{Making Pretrained Language Models Good Long-tailed Learners}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=MH5BazWAeE},
}
Figure