Evaluating Parameter Efficient Learning for Generation

Peng Xu, Mostofa Patwary, Shrimai Prabhumoye, Virginia Adams, Ryan J. Prenger, Wei Ping, Nayeon Lee, Mohammad Shoeybi, Bryan Catanzaro

Abstract: Parameter efficient learning methods (PERMs) have recently gained significant attention as they provide an efficient way for pre-trained language models (PLMs) to adapt to a downstream task. However, these conclusions are mostly drawn from in-domain evaluations over the full training set. In this paper, we present comparisons between PERMs and finetuning from three new perspectives: (1) the effect of sample and model size to in-domain evaluations, (2) generalization to unseen domains and new datasets, and (3) the faithfulness of generations. Our results show that for in-domain settings (a) there is a cross point of sample size for which PERMs will perform better than finetuning when training with fewer samples, and (b) larger PLMs have larger cross points. For cross-domain and cross-dataset cases, we show that (a) Adapter (Houlsby et al., 2019) performs the best amongst all the PERMs studied here, and (b) it outperforms finetuning if the task dataset is below a certain size. We also compare the faithfulness of generations and show that PERMs can achieve better faithfulness score than finetuning, especially for small training set, by as much as 6%. Finally, we apply Adapter to MT-NLG 530b (Smith et al., 2022) and achieve new state-of-the-art results on Xsum (Narayan et al., 2018) for all ROUGE scores (ROUGE-1 49.17, ROUGE-2 27.20, ROUGE-L 40.98).

1. Introduction

Adapting Pre-trained Language Models (PLMs) to specific downstream tasks typically requires full finetuning (FT), where every parameter in the model is updated. For modern Large Language Models (LLMs) with hundreds of billions of parameters, this process is prohibitively expensive and often leads to overfitting, particularly on smaller datasets.

Parameter Efficient Learning Methods (PERMs) offer a solution by freezing the bulk of the pre-trained model and only training a small fraction of additional parameters. While PERMs have shown promise, most research has focused on large, in-domain datasets. This paper investigates the trade-offs between PERMs and FT across scaling, generalization to unseen domains, and the faithfulness of generated text.

2. Methodology

We compare full finetuning against four representative PERMs:

  • Adapters (AP): Small bottleneck layers inserted between existing transformer layers.
  • Prefix Tuning (PF): Trainable "virtual tokens" added to the beginning of transformer block hidden states.
  • Prompt Tuning (PT): Trainable embeddings added only to the input layer.
  • P-tuning: A prompt encoder used to generate pseudo-prompts.

The study primarily focuses on Adapters and Prefix Tuning, as they significantly outperformed Prompt Tuning and P-tuning in generation tasks. We utilized GPT-style models from the Megatron-LM family, ranging from 357M to 530B parameters, evaluated on summarization (Xsum) and dialogue (Wizard of Wikipedia) tasks.

3. The "Cross Point" Phenomenon

A critical finding of this research is the existence of a "cross point"—a specific training set size below which PERMs outperform full finetuning. When data is scarce, the restricted parameter space of PERMs acts as a regularizer, preventing the overfitting common in full finetuning.

Furthermore, as the base model size increases, this cross point moves to the right. For example, on the Xsum dataset, a 1.3B model has a cross point under 10k samples, while an 8.3B model's cross point jumps to approximately 50k samples. This suggests that PERMs become increasingly valuable as models scale.

Figure 1: ROUGE-L and Perplexity comparisons showing the 'cross point' (green stars) where PERMs begin to outperform full finetuning as training data decreases.

4. Generalization and Faithfulness

In cross-domain and cross-dataset evaluations, Adapters demonstrated superior performance compared to Prefix Tuning. This is attributed to the "skip-connection" structure of Adapters, which facilitates smoother optimization and maintains closer proximity to pre-trained weights.

PERMs also exhibit higher "faithfulness" in generation. Full finetuning often forces a model to memorize unfaithful training examples (e.g., summaries containing information not present in the source). Because PERMs have fewer trainable parameters, they are less prone to this memorization, leading to factual consistency scores up to 6% higher than full finetuning.

5. Results and State-of-the-Art

The efficiency of PERMs is substantial: for an 8.3B model, tuning an Adapter with just 32M parameters on 5k samples yielded better results than full finetuning a 357M model on 100k samples. This represents a 90% reduction in task-specific parameters and a 97% reduction in required training data.

Table 1: Performance comparison of Prefix Tuning (PF), Adapters (AP), and Finetuning (FT) across different model sizes and datasets.

By applying Adapters to the MT-NLG 530B model, we achieved a new state-of-the-art on the Xsum benchmark with a ROUGE-L score of 40.98, outperforming specialized models like BRIO and T5-11B.

6. Conclusion

While full finetuning remains the gold standard for very large datasets, PERMs—particularly Adapters—provide a robust and often superior alternative in low-data regimes and for massive-scale models. They offer significant gains in efficiency, generalization, and factual faithfulness, proving that a targeted approach to parameter updates can outperform exhaustive finetuning.

Summary updated

Abstract

Parameter efficient learning methods (PERMs) have recently gained significant attention as they provide an efficient way for pre-trained language models (PLMs) to adapt to a downstream task. However, these conclusions are mostly drawn from in-domain evaluations over the full training set. In this paper, we present comparisons between PERMs and finetuning from three new perspectives: (1) the effect of sample and model size to in-domain evaluations, (2) generalization to unseen domains and new datasets, and (3) the faithfulness of generations. Our results show that for in-domain settings (a) there is a cross point of sample size for which PERMs will perform better than finetuning when training with fewer samples, and (b) larger PLMs have larger cross points. For cross-domain and cross-dataset cases, we show that (a) Adapter (Houlsby et al., 2019) performs the best amongst all the PERMs studied here, and (b) it outperforms finetuning if the task dataset is below a certain size. We also compare the faithfulness of generations and show that PERMs can achieve better faithfulness score than finetuning, especially for small training set, by as much as 6%. Finally, we apply Adapter to MT-NLG 530b (Smith et al., 2022) and achieve new state-of-the-art results on Xsum (Narayan et al., 2018) for all ROUGE scores (ROUGE-1 49.17, ROUGE-2 27.20, ROUGE-L 40.98).
BibTeX
@inproceedings{xu2022evaluating,
  author = {Peng Xu and Mostofa Patwary and Shrimai Prabhumoye and Virginia Adams and Ryan J. Prenger and Wei Ping and Nayeon Lee and Mohammad Shoeybi and Bryan Catanzaro},
  title = {{Evaluating Parameter Efficient Learning for Generation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  doi = {10.18653/v1/2022.emnlp-main.319},
}
Figure