Evaluating Parameter Efficient Learning for Generation

Peng Xu, Mostofa Patwary, Shrimai Prabhumoye, Virginia Adams, Ryan Prenger, Wei Ping, Nayeon Lee, Mohammad Shoeybi, Bryan Catanzaro

Introduction

As Large Language Models (LLMs) scale to hundreds of billions of parameters, such as GPT-3 or MT-NLG, the traditional approach of "full finetuning"—updating every weight in the model—becomes computationally and logistically prohibitive. Parameter Efficient Learning Methods (PERMs), which train only a small fraction (often less than 1%) of parameters, offer a compelling alternative. This paper provides a rigorous comparison between PERMs (like Adapters and Prefix Tuning) and full finetuning (FT) for text generation tasks, uncovering critical insights into when and why PERMs should be preferred.

Methods and Datasets

The researchers evaluated four representative PERMs against a full finetuning baseline using Megatron-LM models ranging from 357M to 530B parameters:

  • Adapters (AP): Trainable bottleneck layers within transformer blocks.
  • Prefix Tuning (PF): Trainable "virtual tokens" added to every layer's hidden states.
  • Prompt Tuning (PT) & P-tuning v2: Trainable embeddings at the input or across all layers.

The study focused on summarization (Xsum, CNN/Daily Mail) and dialogue generation (Wizard of Wikipedia, CMU_DoG) to test both in-domain performance and cross-dataset generalization.

The "Cross Point" Phenomenon

A central finding of the study is the existence of a "cross point": a specific training set size where full finetuning begins to outperform PERMs. In low-data regimes, PERMs are consistently superior because they are less prone to overfitting. Crucially, as the base model size increases, this cross point shifts toward larger datasets. For instance, while the cross point for a 1.3B model on Xsum is approximately 10k samples, it jumps to 50k samples for an 8.3B model.

ROUGE-1 scores across different training set sizes and model scales, illustrating the 'cross point' where full finetuning (FT) begins to outperform parameter-efficient methods.

This suggests that for massive models, PERMs are likely the optimal choice for most practical applications where data is not infinite.

Faithfulness and Generalization

Beyond raw performance metrics like ROUGE, the paper investigates "faithfulness"—the factual accuracy of generated text. Full finetuning often leads models to rely on internal "memorized" knowledge, resulting in hallucinations. By freezing the core model, PERMs force the system to stay grounded in the provided source text. Prefix Tuning, in particular, achieved the highest faithfulness scores, outperforming full finetuning by up to 6% in low-data scenarios.

Furthermore, Adapters emerged as the most robust generalists, consistently outperforming Prefix Tuning in cross-domain and cross-dataset evaluations.

Key Results

The application of Adapters to the massive MT-NLG 530B model yielded a new state-of-the-art on the Xsum benchmark, surpassing significantly larger fully-tuned models.

Comparison of MT-NLG with Adapters against other state-of-the-art models on the Xsum benchmark.

The efficiency gains are equally striking: tuning an Adapter for an 8.3B model with just 5k samples outperformed full finetuning of a 357M model on 100k samples—a 90% reduction in task-specific parameters and a 97% reduction in training data requirements.

Limitations and Future Work

While the existence of cross points is proven, they remain task-dependent, and predicting them for new tasks without empirical testing is still an open challenge. Additionally, a trade-off exists between the superior faithfulness of PERMs and the slightly higher fluency (lower perplexity) sometimes achieved by full finetuning. Future research will focus on bridging this gap to maximize both accuracy and linguistic quality.

Summary updated

BibTeX
@inproceedings{xu2022evaluating,
  author = {Peng Xu and Mostofa Patwary and Shrimai Prabhumoye and Virginia Adams and Ryan Prenger and Wei Ping and Nayeon Lee and Mohammad Shoeybi and Bryan Catanzaro},
  title = {{Evaluating Parameter Efficient Learning for Generation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  doi = {10.18653/v1/2022.emnlp-main.319},
}
Figure