RECOST: External Knowledge Guided Data-efficient Instruction Tuning

Qi Zhang, Yiming Zhang, Haobo Wang, Junbo Zhao

Abstract

Instruction tuning is a critical phase in aligning large language models (LLMs) to follow human directions. However, the high computational cost and the need for high-quality data have led to the rise of "data-efficient" instruction tuning. While synthetic datasets (e.g., Alpaca) are commonly used to reduce human annotation costs, they often contain "dirty" samples—hallucinations or low-quality noise—that traditional selection methods struggle to filter. We introduce RECOST (REtrieval, RE-rank, COreset sampling, and Supervised fine-Tuning), a framework that leverages external knowledge to identify high-quality instructional data. By using a new metric called Relative Predictive Entropy (RPE), RECOST identifies samples where external context significantly reduces model uncertainty. Our experiments show that RECOST can outperform models trained on full datasets using only 1% of the original data.

Introduction: The "Noise Trap" in Synthetic Data

The current paradigm for instruction tuning often relies on synthetic datasets generated by teacher models like GPT-4. While abundant, these datasets are frequently "noisy." A significant challenge in data-efficient tuning is selecting the most "informative" samples. Most existing methods rely on Predictive Entropy—a measure of how uncertain a model is about a given response.

However, the authors identify a "Noise Trap": in synthetic datasets, the samples that elicit the highest uncertainty from a model are often not the most "difficult" or "educational" ones, but rather the most nonsensical or hallucinated ones. Consequently, standard selection methods inadvertently prioritize low-quality noise, leading to suboptimal performance despite reduced data volume.

The RECOST Framework

To overcome the limitations of internal uncertainty metrics, RECOST introduces external validation into the data selection pipeline. The framework consists of four primary stages:

  1. External Knowledge Retrieval: For every instruction-response pair in a synthetic dataset (like Alpaca), the system retrieves relevant factual snippets from a trusted external knowledge base (e.g., Wikipedia or a subset of Flan v2).
  2. Relative Predictive Entropy (RPE): This is the core innovation of the paper. Instead of measuring raw uncertainty, RECOST calculates how much the model's uncertainty decreases when it is provided with the retrieved external snippet as context.
  3. High RPE: If a "hint" from a trusted source significantly clarifies the response for the model, the sample is likely a high-quality, learnable data point that contains factual substance.
  4. Low RPE: If the external hint does not reduce uncertainty, the sample is likely either too trivial or contains "irreducible noise" (hallucinations or gibberish).
  5. Diversity-Consistent Sampling: To ensure the selected subset is not redundant, RECOST employs a modified Core-set sampling algorithm. This ensures that the final 1% of data covers a broad range of topics and tasks while maintaining high RPE scores.
  6. Supervised Fine-Tuning: The final, curated subset is used to fine-tune the base LLM.

Experimental Setup and Results

The researchers evaluated RECOST using the Alpaca (GPT-3.5) and Alpaca-gpt4 datasets, each containing 52,000 samples. They tested the resulting models against the OpenLLM Benchmark and AlpacaEval.

The Power of 1%

The most striking result is the efficiency of the selection process. On the Alpaca dataset, a model trained on just 520 samples (1%) selected by RECOST achieved an AlpacaEval win rate of 39.19%. This significantly outperformed the model trained on the full 52,000 samples (27.75%).

Comparison with State-of-the-Art

RECOST consistently outperformed other popular data-selection methods, such as Instruction-Following Difficulty (IFD). Even when the external knowledge source was changed from the highly structured Flan v2 to raw Wikipedia text, the framework remained effective, demonstrating that RECOST is not overly dependent on a specific reference dataset.

Key Insights

  • Sensitivity as a Quality Signal: A model’s sensitivity to external truth is a more reliable indicator of a sample's value than its internal uncertainty.
  • Retrieval-Augmented Selection: While Retrieval-Augmented Generation (RAG) is typically used during inference, RECOST demonstrates its power as a "data cleaner" during the training phase.
  • Less is More: Following the philosophy of the LIMA paper, RECOST proves that alignment is more about the quality and diversity of the examples than the sheer volume of data, especially when dealing with synthetic sources.

Limitations and Future Work

While highly effective, RECOST introduces additional computational overhead during the pre-processing phase due to the retrieval and dual-entropy calculations. Furthermore, the method's performance is tied to the quality of the retriever; if the knowledge base does not contain information relevant to a specific instruction, the RPE metric may not accurately reflect the sample's quality. Future research will focus on optimizing these pre-processing steps and exploring more diverse external knowledge sources to further refine the selection process.

Summary updated

Abstract

In the current landscape of large language models (LLMs), the process of instruction tuning serves as an essential step. Considering the high computing power overhead, data-efficient instruction tuning was proposed to reduce the training data size in this process, aiming at selecting high-quality instructional data. Nevertheless, we argue that most current data-efficient instruction-tuning methods are highly dependent on the quality of the original instruction-tuning dataset. When it comes to datasets synthesized by LLMs, a common scenario in this field, dirty samples will even be selected with a higher probability than other samples. To address these challenges, we utilized external knowledge (relevant examples or paragraphs) to evaluate those samples synthesized by LLMs with an in-context-based relative predictive entropy. Based on the new metric, we proposed a framework, dubbed as \textbf{RECOST}, which integrates external-knowledge-base re-ranking and diversity-consistent sampling into a single pipeline. Through extensive experiments on several synthetic datasets (Alpaca and Alpaca-gpt4), we demonstrate the effectiveness of our method and achieve even better results with only \textbf{1\%} of the full dataset.
BibTeX
@inproceedings{zhang2024recost,
  author = {Qi Zhang and Yiming Zhang and Haobo Wang and Junbo Zhao},
  title = {{RECOST: External Knowledge Guided Data-efficient Instruction Tuning}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  url = {https://openreview.net/forum?id=CteYJAEEen},
}
Figure