Abstract
Retrieval-augmented models are typically designed to leverage massive external corpora to handle knowledge-intensive tasks. However, these methods often incur significant computational costs and may suffer from distributional misalignment. This paper introduces REINA (REtrieving from the traINing datA), a simple yet powerful approach that treats the training set itself as the primary knowledge base. By retrieving similar labeled examples from the training data and concatenating them with the input at inference time, REINA achieves state-of-the-art performance across summarization, language modeling, machine translation, and question answering, often allowing smaller models to outperform significantly larger baselines.
Introduction
In Natural Language Processing (NLP), Large Language Models (LLMs) are expected to internalize vast amounts of knowledge during training. While effective, even models with hundreds of millions of parameters struggle to memorize every nuance. Existing solutions often rely on "retrieval-augmentation," fetching external knowledge from sources like Wikipedia. While powerful, these methods are frequently expensive to index, slow during inference (e.g., kNN-MT), and may not align with the specific style or format of the target task.
We argue that the training data is the most relevant and distributionally aligned knowledge base available, yet it is typically discarded after the training phase. REINA addresses this by using the training data as a "cheat sheet" for the model during both training and inference.
Methods
The REINA pipeline is designed for simplicity and efficiency. It consists of four primary stages: indexing, retrieval, filtering, and concatenation.
- Index: Labeled training data is stored as input-label pairs.
- Retrieve: For a given input, the system employs BM25, a fast sparse retrieval algorithm, to identify the $K$ most similar training instances. BM25 is chosen for its speed and ability to handle long documents without the need for a separate dense encoder.
- Filter: To prevent data leakage during the training phase, the specific instance being processed is filtered from the retrieved results.
- Concatenate: The retrieved examples are prepended or appended to the input. This augmented input is then fed into a standard transformer model (e.g., BART, PEGASUS, or DeBERTa) for final prediction.
Experimental Results
REINA was evaluated across 11 datasets spanning four major NLP categories. The method demonstrated consistent improvements, often reaching new state-of-the-art (SOTA) levels.
Summarization
On summarization benchmarks, REINA showed significant gains. Notably, on the BigPatent dataset, REINA (Large) achieved a Rouge-1 score of 60.7, a substantial improvement over previous SOTA models. On XSum, the model reached 48.2 Rouge-1. A key finding is that BART-base with REINA rivals BART-large on several tasks, effectively doubling the model's effective knowledge without increasing parameter count.
Language Modeling and Question Answering
For language modeling on WikiText103, REINA reduced perplexity from 15.88 to 14.76 for BART-base. In question answering, REINA achieved the No. 1 position on the CommonsenseQA leaderboard at the time of publication, with a test accuracy of 84.6%.
Discussion and Insights
The success of REINA highlights the "Cheat Sheet" effect: even if a model has seen a sample during training, presenting it again as a side-by-side reminder helps it recapture complex patterns that were not fully internalized in its weights. Furthermore, because training data is by definition the closest possible distribution to the test data, it serves as a more reliable source of truth than general-purpose external corpora.
Conclusion
REINA demonstrates that training data is far more valuable than previously thought. By simply retrieving and concatenating similar training examples, models can achieve significant performance boosts with minimal overhead. This suggests that before seeking external data, researchers should prioritize making better use of the data already at hand.
Summary updated