Training Data is More Valuable than You Think: A Simple and Effective Method by Retrieving from Training Data

Shuohang Wang, Yichong Xu, Yuwei Fang, Yang Liu, Siqi Sun, Ruochen Xu, Chenguang Zhu, Michael Zeng

Abstract

Retrieval-augmented models are typically designed to leverage massive external corpora to handle knowledge-intensive tasks. However, these methods often incur significant computational costs and may suffer from distributional misalignment. This paper introduces REINA (REtrieving from the traINing datA), a simple yet powerful approach that treats the training set itself as the primary knowledge base. By retrieving similar labeled examples from the training data and concatenating them with the input at inference time, REINA achieves state-of-the-art performance across summarization, language modeling, machine translation, and question answering, often allowing smaller models to outperform significantly larger baselines.

Introduction

In Natural Language Processing (NLP), Large Language Models (LLMs) are expected to internalize vast amounts of knowledge during training. While effective, even models with hundreds of millions of parameters struggle to memorize every nuance. Existing solutions often rely on "retrieval-augmentation," fetching external knowledge from sources like Wikipedia. While powerful, these methods are frequently expensive to index, slow during inference (e.g., kNN-MT), and may not align with the specific style or format of the target task.

We argue that the training data is the most relevant and distributionally aligned knowledge base available, yet it is typically discarded after the training phase. REINA addresses this by using the training data as a "cheat sheet" for the model during both training and inference.

Methods

The REINA pipeline is designed for simplicity and efficiency. It consists of four primary stages: indexing, retrieval, filtering, and concatenation.

The REINA pipeline: Indexing, Retrieving, Filtering, and Concatenating.
  1. Index: Labeled training data is stored as input-label pairs.
  2. Retrieve: For a given input, the system employs BM25, a fast sparse retrieval algorithm, to identify the $K$ most similar training instances. BM25 is chosen for its speed and ability to handle long documents without the need for a separate dense encoder.
  3. Filter: To prevent data leakage during the training phase, the specific instance being processed is filtered from the retrieved results.
  4. Concatenate: The retrieved examples are prepended or appended to the input. This augmented input is then fed into a standard transformer model (e.g., BART, PEGASUS, or DeBERTa) for final prediction.

Experimental Results

REINA was evaluated across 11 datasets spanning four major NLP categories. The method demonstrated consistent improvements, often reaching new state-of-the-art (SOTA) levels.

Summarization

On summarization benchmarks, REINA showed significant gains. Notably, on the BigPatent dataset, REINA (Large) achieved a Rouge-1 score of 60.7, a substantial improvement over previous SOTA models. On XSum, the model reached 48.2 Rouge-1. A key finding is that BART-base with REINA rivals BART-large on several tasks, effectively doubling the model's effective knowledge without increasing parameter count.

Summarization results across various datasets showing REINA outperforming strong baselines like PEGASUS and BART.

Language Modeling and Question Answering

For language modeling on WikiText103, REINA reduced perplexity from 15.88 to 14.76 for BART-base. In question answering, REINA achieved the No. 1 position on the CommonsenseQA leaderboard at the time of publication, with a test accuracy of 84.6%.

Discussion and Insights

The success of REINA highlights the "Cheat Sheet" effect: even if a model has seen a sample during training, presenting it again as a side-by-side reminder helps it recapture complex patterns that were not fully internalized in its weights. Furthermore, because training data is by definition the closest possible distribution to the test data, it serves as a more reliable source of truth than general-purpose external corpora.

Conclusion

REINA demonstrates that training data is far more valuable than previously thought. By simply retrieving and concatenating similar training examples, models can achieve significant performance boosts with minimal overhead. This suggests that before seeking external data, researchers should prioritize making better use of the data already at hand.

Summary updated

Abstract

Retrieval-based methods have been shown to be effective in NLP tasks via introducing external knowledge. However, the indexing and retrieving of large-scale corpora bring considerable computational cost. Surprisingly, we found that REtrieving from the traINing datA (REINA) only can lead to significant gains on multiple NLG and NLU tasks. We retrieve the labeled training instances most similar to the input text and then concatenate them with the input to feed into the model to generate the output. Experimental results show that this simple method can achieve significantly better performance on a variety of NLU and NLG tasks, including summarization, machine translation, language modeling, and question answering tasks. For instance, our proposed method achieved state-of-the-art results on XSum, BigPatent, and CommonsenseQA. Our code is released, https://github.com/microsoft/REINA .
BibTeX
@inproceedings{wang2022training,
  author = {Shuohang Wang and Yichong Xu and Yuwei Fang and Yang Liu and Siqi Sun and Ruochen Xu and Chenguang Zhu and Michael Zeng},
  title = {{Training Data is More Valuable than You Think: A Simple and Effective Method by Retrieving from Training Data}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.18653/v1/2022.acl-long.226},
}
Figure