Selective Reflection-Tuning: Student-Selected Data Recycling for LLM Instruction-Tuning

Ming Li, Lichang Chen, Jiuhai Chen, Shwai He, Jiuxiang Gu, Tianyi Zhou

Abstract

Instruction tuning is critical to large language models (LLMs) for achieving better instruction following and task adaptation capabilities, but its success heavily relies on the training data quality. Many recent methods focus on improving the data quality but often overlook the compatibility of the data with the student model being finetuned. This paper introduces Selective Reflection-Tuning, a novel paradigm that synergizes a teacher LLM's reflection and introspection for improving existing data quality with the data selection capability of the student LLM, to automatically refine existing instruction-tuning data. This teacher-student collaboration produces high-quality and student-compatible instruction-response pairs, resulting in sample-efficient instruction tuning and LLMs of superior performance. Selective Reflection-Tuning is a data augmentation and synthesis method that generally improves LLM finetuning and self-improvement without collecting brand-new data. We apply our method to Alpaca and WizardLM data and achieve much stronger and top-tier 7B and 13B LLMs.

Introduction: The Compatibility Gap

Instruction tuning—the process of fine-tuning Large Language Models (LLMs) to follow specific prompts—is only as good as the data used to train it. Traditionally, researchers have relied on human experts to curate datasets or used "teacher" models (like GPT-4) to generate massive amounts of synthetic data.

However, a significant "compatibility gap" often persists. A teacher model might produce an "improved" instruction that is too complex, too random, or simply misaligned with the student model’s current capabilities. Most existing methods are teacher-dominant: the teacher decides what is "good," and the student is forced to learn it. Selective Reflection-Tuning (SRT) addresses this by asking: How can the student model autonomously decide which data is actually critical to its own training?

Methodology: Selective Reflection-Tuning

The authors introduce a teacher-student collaboration pipeline that happens in two main phases, focusing on difficulty and feasibility.

The Selective Reflection-Tuning pipeline overview.

1. Selective Instruction Reflection

In this phase, a teacher model (e.g., ChatGPT) reflects on an existing instruction and generates a more complex, difficult version. The student model then calculates an Instruction-Following Difficulty (IFD) score for both the old and new versions. The student keeps the version with the higher IFD score, essentially choosing the more challenging "lesson" that pushes its current boundaries.

2. Selective Response Reflection

The teacher then improves the response for the selected instruction. To ensure this new response is actually helpful, the authors introduce a new metric: reversed-IFD (r-IFD). This measures how easily the student can "guess" the instruction given the response. If the r-IFD is low, the sample is considered "feasible"—meaning the response is clear and logically connected to the prompt. The student selects the version with the lower r-IFD, ensuring the data is learnable rather than just complex.

Experimental Setup and Datasets

The researchers applied SRT to two widely used open-source datasets: Alpaca (originally 52,002 samples) and WizardLM (a subset of 70,000 samples). The resulting "sRecycled" models were evaluated across three major benchmarks:

  • AlpacaEval Leaderboard: A simulator for human preferences.
  • Huggingface Open LLM Leaderboard: A battery of reasoning and knowledge tasks (ARC, HellaSwag, MMLU, etc.).
  • MT-Bench: A benchmark for multi-turn conversation capabilities.

Key Results and Efficiency

The results demonstrate that quality beats quantity by a landslide. The most significant finding is the extreme sample efficiency of the SRT approach.

Alpaca Eval Win Rate vs. Data Used, highlighting the efficiency of sRecycled models.
  • Efficiency: The "sRecycled WizardLM 7B" model, trained on just 1,000 samples (2% of the original data), outperformed the original WizardLM 7B and even the LIMA model, which relied on 1,000 manually curated human samples.
  • Top-Tier Performance: The sRecycled WizardLM 13B achieved a win rate of 85.96% on the AlpacaEval Leaderboard, outperforming much larger models and those trained on significantly more data, such as the 65B Humpback LLaMA.
  • Consistency: Across the board, the "sRecycled" versions of Alpaca and WizardLM showed marked improvements over their base versions in reasoning (ARC), common sense (HellaSwag), and general knowledge (MMLU).

Discussion and Insights

The core "aha moment" of this research is that data quality is subjective to the model being trained. A data point that helps a 70B model might be "noise" to a 7B model. By using the student's own perplexity scores to filter data, the training becomes much more efficient. Furthermore, while previous work focused solely on finding difficult data, this paper highlights feasibility. If a response is so complex that the model cannot associate it back to the original instruction (High r-IFD), it is likely to cause degradation rather than improvement.

Limitations and Future Work

Because the IFD and r-IFD scores are derived from the student model itself, they must be recalculated for every new architecture. Additionally, calculating perplexity for every sample adds computational overhead, though this is arguably offset by the reduced training time on smaller, high-quality datasets. Future work will look to extend this "selective reflection" to complex, multi-turn dialogues.

Summary updated

Abstract

Instruction tuning is critical to large language models (LLMs) for achieving better instruction following and task adaptation capabilities but its success heavily relies on the training data quality. Many recent methods focus on improving the data quality but often overlook the compatibility of the data with the student model being finetuned. This paper introduces Selective Reflection-Tuning, a novel paradigm that synergizes a teacher LLM's reflection and introspection for improving existing data quality with the data selection capability of the student LLM, to automatically refine existing instruction-tuning data. This teacher-student collaboration produces high-quality and student-compatible instruction-response pairs, resulting in sample-efficient instruction tuning and LLMs of superior performance. Selective Reflection-Tuning is a data augmentation and synthesis that generally improves LLM finetuning and self-improvement without collecting brand-new data. We apply our method to Alpaca and WizardLM data and achieve much stronger and top-tier 7B and 13B LLMs.
BibTeX
@inproceedings{li2024selective,
  author = {Ming Li and Lichang Chen and Jiuhai Chen and Shwai He and Jiuxiang Gu and Tianyi Zhou},
  title = {{Selective Reflection-Tuning: Student-Selected Data Recycling for LLM Instruction-Tuning}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  url = {https://openreview.net/forum?id=nvXucRIPJ0},
}

Loading related content...

Figure