Abstract
This study investigates the synergy between Retrieval-Augmented Generation (RAG) and Parameter-Efficient Fine-Tuning (PEFT) by comparing standard GPT architectures against Retrieval-Enhanced Transformer (RETRO) models. While RETRO models demonstrate superior performance in zero-shot settings due to their retrieval-centric pre-training, standard GPT models exhibit a higher performance ceiling when fine-tuned using PEFT techniques. Our analysis identifies 8B parameter models as the optimal "sweet spot" for balancing computational cost and task accuracy.
Introduction
Adapting Large Language Models (LLMs) to specialized domains typically follows two paths: Retrieval-Augmented Generation (RAG), which provides external knowledge at inference, and Parameter-Efficient Fine-Tuning (PEFT), which updates a small subset of weights for task-specific behavior. Despite their popularity, these approaches are rarely studied in tandem across different architectures. This paper explores the impact of applying PEFT to models specifically pre-trained for retrieval (RETRO) versus standard decoder-only models (GPT) across scales ranging from 823M to 48B parameters.
Methodology
The research compares two primary architectures: a standard auto-regressive GPT baseline and RETRO, which integrates retrieved context via chunked cross-attention. Three PEFT techniques were evaluated:
- LoRA (Low-Rank Adaptation): Injecting trainable low-rank matrices into existing layers.
- Adapters: Inserting small, fully connected modules between layers.
- P-tuning: Training continuous virtual tokens prepended to the input.
Retrieval was facilitated by Dragon+, a state-of-the-art dual encoder, across six diverse datasets including Natural Questions (NQ), TriviaQA (TQA), and long-document tasks like QuALITY and QMSum.
Scaling and Performance Insights
One of the most significant findings is the performance trajectory as models scale. RETRO models consistently win the "cold start" in zero-shot scenarios because they are architecturally designed to utilize external context immediately. However, as fine-tuning is introduced, GPT models often overtake RETRO. This suggests that GPT models have a higher "learning ceiling"; since they were not retrieval-focused during pre-training, PEFT allows them to make more substantial gains in context utilization.
Both architectures show diminishing returns beyond the 8B parameter mark, establishing it as the practical "sweet spot" for most applications. Notably, P-tuning performed significantly worse on RETRO, likely because it only affects the decoder, whereas LoRA and Adapters were applied to both the decoder and the retrieval encoder, allowing for better end-to-end integration.
Key Results: The Fine-Tuning Flip
At the largest scales tested (43B–48B parameters), the performance gap between GPT and RETRO becomes most apparent. In zero-shot settings, the Extra-Small RETRO (877M) outperformed its GPT counterpart (8.58 vs 6.54). However, at the Extra-Large scale, GPT + LoRA achieved an average score of 43.72, significantly surpassing RETRO + LoRA at 37.41.
Furthermore, the study found that applying PEFT to already instruction-tuned RETRO models (I-RETRO) yielded only marginal gains, suggesting that instruction tuning may saturate the model's capacity for further improvement via parameter-efficient methods.
Conclusion and Future Work
While RETRO provides a strong baseline for retrieval tasks out of the box, standard GPT models paired with PEFT offer superior peak performance at scale. Future research should investigate whether these trends hold for "frontier" models exceeding 100B parameters and explore novel PEFT methods specifically designed to target the unique cross-attention mechanisms in retrieval-heavy architectures.
Summary updated