GPT vs RETRO: Exploring the Intersection of Retrieval and Parameter-Efficient Fine-Tuning

Aleksander Ficek, Jiaqi Zeng, Oleksii Kuchaiev

Abstract

This study investigates the synergy between Retrieval-Augmented Generation (RAG) and Parameter-Efficient Fine-Tuning (PEFT) by comparing standard GPT architectures against Retrieval-Enhanced Transformer (RETRO) models. While RETRO models demonstrate superior performance in zero-shot settings due to their retrieval-centric pre-training, standard GPT models exhibit a higher performance ceiling when fine-tuned using PEFT techniques. Our analysis identifies 8B parameter models as the optimal "sweet spot" for balancing computational cost and task accuracy.

Introduction

Adapting Large Language Models (LLMs) to specialized domains typically follows two paths: Retrieval-Augmented Generation (RAG), which provides external knowledge at inference, and Parameter-Efficient Fine-Tuning (PEFT), which updates a small subset of weights for task-specific behavior. Despite their popularity, these approaches are rarely studied in tandem across different architectures. This paper explores the impact of applying PEFT to models specifically pre-trained for retrieval (RETRO) versus standard decoder-only models (GPT) across scales ranging from 823M to 48B parameters.

Methodology

The research compares two primary architectures: a standard auto-regressive GPT baseline and RETRO, which integrates retrieved context via chunked cross-attention. Three PEFT techniques were evaluated:

  • LoRA (Low-Rank Adaptation): Injecting trainable low-rank matrices into existing layers.
  • Adapters: Inserting small, fully connected modules between layers.
  • P-tuning: Training continuous virtual tokens prepended to the input.

Retrieval was facilitated by Dragon+, a state-of-the-art dual encoder, across six diverse datasets including Natural Questions (NQ), TriviaQA (TQA), and long-document tasks like QuALITY and QMSum.

Scaling and Performance Insights

One of the most significant findings is the performance trajectory as models scale. RETRO models consistently win the "cold start" in zero-shot scenarios because they are architecturally designed to utilize external context immediately. However, as fine-tuning is introduced, GPT models often overtake RETRO. This suggests that GPT models have a higher "learning ceiling"; since they were not retrieval-focused during pre-training, PEFT allows them to make more substantial gains in context utilization.

Average scores across model scales for GPT (top) and RETRO (bottom) using various PEFT methods.

Both architectures show diminishing returns beyond the 8B parameter mark, establishing it as the practical "sweet spot" for most applications. Notably, P-tuning performed significantly worse on RETRO, likely because it only affects the decoder, whereas LoRA and Adapters were applied to both the decoder and the retrieval encoder, allowing for better end-to-end integration.

Key Results: The Fine-Tuning Flip

At the largest scales tested (43B–48B parameters), the performance gap between GPT and RETRO becomes most apparent. In zero-shot settings, the Extra-Small RETRO (877M) outperformed its GPT counterpart (8.58 vs 6.54). However, at the Extra-Large scale, GPT + LoRA achieved an average score of 43.72, significantly surpassing RETRO + LoRA at 37.41.

Comparison of Extra Large GPT vs. RETRO average scores across different fine-tuning methods.

Furthermore, the study found that applying PEFT to already instruction-tuned RETRO models (I-RETRO) yielded only marginal gains, suggesting that instruction tuning may saturate the model's capacity for further improvement via parameter-efficient methods.

Conclusion and Future Work

While RETRO provides a strong baseline for retrieval tasks out of the box, standard GPT models paired with PEFT offer superior peak performance at scale. Future research should investigate whether these trends hold for "frontier" models exceeding 100B parameters and explore novel PEFT methods specifically designed to target the unique cross-attention mechanisms in retrieval-heavy architectures.

Summary updated

Abstract

Parameter-Efficient Fine-Tuning (PEFT) and Retrieval-Augmented Generation (RAG) have become popular methods for adapting large language models while minimizing compute requirements. In this paper, we apply PEFT methods (P-tuning, Adapters, and LoRA) to a modified Retrieval-Enhanced Transformer (RETRO) and a baseline GPT model across several sizes, ranging from 823 million to 48 billion parameters. We show that RETRO models outperform GPT models in zero-shot settings due to their unique pre-training process but GPT models have higher performance potential with PEFT. Additionally, our study indicates that 8B parameter models strike an optimal balance between cost and performance and P-tuning lags behind other PEFT techniques. We further provide a comparative analysis between applying PEFT to an Instruction-tuned RETRO model and base RETRO model. This work presents the first comprehensive comparison of various PEFT methods integrated with RAG, applied to both GPT and RETRO models, highlighting their relative performance.
BibTeX
@inproceedings{ficek2024gpt,
  author = {Aleksander Ficek and Jiaqi Zeng and Oleksii Kuchaiev},
  title = {{GPT vs RETRO: Exploring the Intersection of Retrieval and Parameter-Efficient Fine-Tuning}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.1081},
}
Figure