RECOST: External Knowledge Guided Data-efficient Instruction Tuning

Qi Zhang, Yiming Zhang, Haobo Wang, Junbo Zhao

Abstract

In the current landscape of large language models (LLMs), the process of instruction tuning serves as an essential step. Considering the high computing power overhead, data-efficient instruction tuning was proposed to reduce the training data size in this process, aiming at selecting high-quality instructional data. Nevertheless, we argue that most current data-efficient instruction-tuning methods are highly dependent on the quality of the original instruction-tuning dataset. When it comes to datasets synthesized by LLMs, a common scenario in this field, dirty samples will even be selected with a higher probability than other samples. To address these challenges, we utilized external knowledge (relevant examples or paragraphs) to evaluate those samples synthesized by LLMs with an in-context-based relative predictive entropy. Based on the new metric, we proposed a framework, dubbed as RECOST, which integrates external-knowledge-base re-ranking and diversity-consistent sampling into a single pipeline. Through extensive experiments on several synthetic datasets (Alpaca and Alpaca-gpt4), we demonstrate the effectiveness of our method and achieve even better results with only 1% of the full dataset.

Introduction

Instruction tuning—fine-tuning a pre-trained LLM on (instruction, response) pairs—is essential for making models follow human commands. However, training on massive datasets is computationally expensive. While "data-efficient" methods exist to select the "best" subsets, they often rely on predictive entropy (a measure of model uncertainty).

This paper identifies a critical flaw: in synthetic datasets like Alpaca, high uncertainty often signals noisy or hallucinated data rather than "difficult but useful" data. Standard metrics accidentally prioritize these "dirty" samples, leading to sub-optimal tuning. The challenge lies in distinguishing between a sample that is genuinely hard to learn and one that is simply incorrect.

The "Entropy Trap" in Synthetic Data

Traditional data selection often relies on metrics like predictive entropy (PE) or Instruction-Following Difficulty (IFD) to identify high-value samples. However, in synthetic datasets, high uncertainty often correlates with "dirty" data.

Comparison of dirty data hit rates between different metrics. RECOST significantly reduces the selection of noisy samples in the top-ranked data.

As shown in the figure above, standard metrics like IFD and predictive entropy have high "dirty data hit rates" among the top-ranked samples. RECOST addresses this by using external knowledge to distinguish between genuine difficulty and noise, effectively filtering out hallucinations.

The RECOST Framework

RECOST (REtrieval, RE-rank, COreset sampling, and Supervised fine-Tuning) introduces a novel metric: Relative Predictive Entropy (RPE). The framework operates in three primary stages:

  1. External Knowledge Retrieval: For each synthetic instruction, relevant "truthful" snippets are retrieved from external sources like Flan v2 or Wikipedia.
  2. Relative Predictive Entropy (RPE): The model's uncertainty is measured both with and without the retrieved knowledge. RPE is defined as the difference: $RPE = PE - PE_{ic}$. A high RPE indicates that the model's uncertainty drops significantly when provided with factual context, suggesting the sample is grounded and learnable.
  3. Diversity-Consistent Sampling: To ensure broad task coverage, a modified Core-set sampling algorithm with a sliding window is applied to the re-ranked data.
The RECOST framework pipeline, highlighting the external-knowledge-based re-ranking and diversity-consistent sampling stages.

Experimental Results

The effectiveness of RECOST was validated on the Alpaca and Alpaca-gpt4 datasets. Remarkably, using only 1% (520 samples) of the Alpaca dataset, RECOST achieved an AlpacaEval win rate of 39.19%, significantly outperforming the model trained on the 100% (52k samples) full dataset (27.75%). It also surpassed other state-of-the-art selection methods like IFD, which required more data (6%) to reach a lower win rate (36.78%).

Even when using Wikipedia instead of Flan v2 as the knowledge source, the method remained highly effective, proving it isn't strictly dependent on one specific "gold" dataset.

Conclusion

RECOST demonstrates that for instruction tuning, quality and factual grounding are far more important than quantity. By leveraging external knowledge to filter out synthetic noise, RECOST provides a robust and highly efficient path for training high-performing LLMs with minimal data. This automated approach reinforces the hypothesis that a small amount of high-quality data can outperform massive, noisy datasets.

Summary updated

Abstract

In the current landscape of large language models (LLMs), the process of instruction tuning serves as an essential step. Considering the high computing power overhead, data-efficient instruction tuning was proposed to reduce the training data size in this process, aiming at selecting high-quality instructional data. Nevertheless, we argue that most current data-efficient instruction-tuning methods are highly dependent on the quality of the original instruction-tuning dataset. When it comes to datasets synthesized by LLMs, a common scenario in this field, dirty samples will even be selected with a higher probability than other samples. To address these challenges, we utilized external knowledge (relevant examples or paragraphs) to evaluate those samples synthesized by LLMs with an in-context-based relative predictive entropy. Based on the new metric, we proposed a framework, dubbed as RECOST, which integrates external-knowledge-base re-ranking and diversity-consistent sampling into a single pipeline. Through extensive experiments on several synthetic datasets (Alpaca and Alpaca-gpt4), we demonstrate the effectiveness of our method and achieve even better results with only 1% of the full dataset.
BibTeX
@inproceedings{zhang2024recost,
  author = {Qi Zhang and Yiming Zhang and Haobo Wang and Junbo Zhao},
  title = {{RECOST: External Knowledge Guided Data-efficient Instruction Tuning}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  url = {https://openreview.net/forum?id=CteYJAEEen},
}
Figure