An Empirical Study on the Transferability of Transformer Modules in Parameter-efficient Fine-tuning

Mohammad Akbar-Tajari, Sara Rajaee, Mohammad Taher PilehvarEMNLP 2022 · DOI

Introduction

As Large Language Models (LLMs) grow in scale, the standard practice of "Full Fine-Tuning"—updating all model parameters for downstream tasks—becomes increasingly prohibitive due to high computational and storage costs. Parameter-Efficient Fine-Tuning (PEFT) has emerged as a viable alternative, seeking to achieve comparable performance by modifying only a small subset of parameters. This paper, presented at EMNLP 2022, investigates the internal mechanics of Transformer architectures to identify which specific modules serve as the most effective "levers" for task adaptation.

Methodology

The researchers conducted a systematic module-wise ablation study using BERT-base (110M parameters). By freezing the majority of the network and isolating specific components for tuning, they evaluated the transferability of:

  • Multi-Head Attention (MHA): The core mechanism for relating tokens.
  • Feed-Forward Networks (FFN): Dense layers processing token representations.
  • Layer Normalization (LayerNorm): Modules that normalize inputs to stabilize training.
  • BitFit: A baseline tuning only bias terms.

The study evaluated performance across the GLUE benchmark for sentence-level tasks and datasets like CoNLL-2003 for token-level tasks such as Named Entity Recognition (NER).

The LayerNorm "Winning Ticket"

A surprising finding of the study is that LayerNorm modules—which comprise a mere 0.03% of BERT's total parameters—are exceptionally powerful vehicles for transfer learning. Tuning only these modules often yields performance nearly identical to full fine-tuning.

The authors attribute this effectiveness to the unique weight distribution within LayerNorms. Unlike the unimodal distributions found in MHA or FFN layers, LayerNorm weights exhibit a distinct bimodal distribution.

Figure 1: Weight distributions of different Transformer modules. LayerNorms show a bimodal distribution with high-magnitude outliers that are crucial for transferability.

These high-magnitude "outlier" weights appear to act as a "winning ticket." The researchers found that tuning just a few hundred of these outlier parameters (as few as 4 to 256 per module) significantly outperformed tuning a random selection of weights of the same size. This suggests that the pre-training process concentrates transferable knowledge into these specific dimensions.

Performance Analysis

On the GLUE benchmark, LayerNorm tuning achieved an average score of 80.6, remarkably close to the 82.8 achieved by Full Fine-Tuning, despite using 3,333x fewer parameters.

Figure 2: Performance comparison of different fine-tuning strategies on the GLUE benchmark. LayerNorm tuning achieves competitive results with a fraction of the parameters.

As shown in the results, LayerNorm tuning (0.03% parameters) outperformed tuning the much larger MHA (25.89%) or FFN (51.76%) modules on several specific tasks. It also proved more efficient than BitFit (0.12%), which requires four times as many parameters to achieve similar performance. Layer-wise analysis further indicated that LayerNorms in the middle of the 12-layer stack are the most critical for adaptation, likely because these layers capture the most versatile linguistic features.

Conclusion

This research demonstrates that the structural consistency of tuning specific functional modules—particularly LayerNorms—is more important than the absolute number of parameters tuned. The discovery of "outlier" weights as primary drivers of transferability suggests new avenues for even more minimalist PEFT methods that target specific high-impact weights rather than entire layers. Future work may explore whether these findings generalize to larger decoder-only architectures like GPT-4 or Llama.

Summary updated

BibTeX
@inproceedings{akbartajari2022empirical,
  author = {Mohammad Akbar-Tajari and Sara Rajaee and Mohammad Taher Pilehvar},
  title = {{An Empirical Study on the Transferability of Transformer Modules in Parameter-efficient Fine-tuning}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  doi = {10.18653/v1/2022.emnlp-main.726},
}
Figure