Selective Reflection-Tuning: Student-Selected Data Recycling for LLM Instruction-Tuning

Ming Li, Lichang Chen, Jiuhai Chen, Shwai He, Jiuxiang Gu, Tianyi Zhou

Abstract

Instruction tuning is critical to large language models (LLMs) for achieving better instruction following and task adaptation capabilities, but its success heavily relies on the training data quality. Many recent methods focus on improving the data quality but often overlook the compatibility of the data with the student model being finetuned. This paper introduces Selective Reflection-Tuning, a novel paradigm that synergizes a teacher LLM’s reflection and introspection for improving existing data quality with the data selection capability of the student LLM, to automatically refine existing instruction-tuning data. This teacher-student collaboration produces high-quality and student-compatible instruction-response pairs, resulting in sample-efficient instruction tuning and LLMs of superior performance. Selective Reflection-Tuning is a data augmentation and synthesis method that generally improves LLM finetuning and self-improvement without collecting brand-new data. We apply our method to Alpaca and WizardLM data and achieve much stronger and top-tier 7B and 13B LLMs.

Introduction

Instruction tuning—the process of fine-tuning a base LLM to follow prompts—is only as effective as the data used to train it. While powerful "teacher" models (like GPT-4) are frequently used to generate or improve datasets, these methods often suffer from a compatibility gap. A teacher model might produce an "improved" instruction that is too complex or simply doesn't align with the student model's internal distribution.

Most current pipelines are "teacher-dominating," where the student is forced to learn from whatever the teacher provides. Selective Reflection-Tuning introduces a "teacher-student collaboration" where the teacher improves existing data, but the student LLM (the one being trained) acts as its own "admissions officer." By selecting data that is both challenging and feasible for its specific architecture, a model can achieve superior performance with significantly fewer samples.

Methodology

The Selective Reflection-Tuning process consists of a three-phase pipeline designed to ensure data is "model-compatible" by using the student's own statistical attributes (perplexity) to filter teacher outputs.

The three-phase Selective Reflection-Tuning pipeline, illustrating the collaboration between Teacher and Student models.

1. Selective Instruction Reflection

A teacher model reflects on an existing instruction to make it more complex and detailed. The student model then calculates the Instruction-Following Difficulty (IFD) score for both the original and the new instruction. It keeps the version with the higher IFD score, as a higher score indicates the instruction provides a more challenging and useful learning signal for that specific model.

2. Selective Response Reflection

The teacher model improves the response for the selected instruction. The student evaluates this using a novel metric: reversed-IFD (r-IFD). This measures how easily the student can "guess" the instruction given the response. A lower r-IFD score indicates the response is highly relevant and feasible for the student to learn from.

3. Instruction Tuning

The final "recycled" dataset—consisting of samples the student model itself deemed high-quality and compatible—is used for standard fine-tuning. This approach tailors the curriculum to the model's size and capability.

Experimental Results

The researchers applied their method to the Alpaca (52k samples) and WizardLM (70k samples) datasets. Evaluation was conducted using the AlpacaEval Leaderboard, the Huggingface Open LLM Leaderboard, and MT-Bench.

Performance and Sample Efficiency

As shown in the leaderboard below, the "sRecycled WizardLM 13B" achieved a 85.96% win rate on AlpacaEval, outperforming much larger models and even those trained with Reinforcement Learning from Human Feedback (RLHF).

Alpaca Eval Leaderboard showing the performance of sRecycled models compared to industry baselines.

Key findings include:

  • Extreme Sample Efficiency: In a "Fewer Data Scenario," using just 926 high-quality samples (2% of the dataset) selected via this method allowed a 7B model to outperform the original LIMA model, which was trained on 1,000 manually curated samples.
  • Response Quality > Instruction Complexity: Ablation studies showed that reflecting on the response (making it more detailed and accurate) had a larger impact on performance than simply increasing instruction complexity.
  • The "Less is More" Paradigm: Performance actually decreased on some benchmarks as data volume increased beyond a certain point, suggesting that "data recycling" (improving a small set of data) is more effective than gathering massive amounts of mediocre data.

Limitations and Future Work

While highly effective, Selective Reflection-Tuning involves computational overhead because IFD and r-IFD scores must be re-calculated for every new model architecture. Additionally, the current study focused primarily on single-turn instructions. While 1-turn scores on MT-Bench were excellent, multi-turn performance remains an area for future adaptation. Finally, the quality of the recycled data remains dependent on the teacher model's ability to provide meaningful initial reflections.

Summary updated

Abstract

Instruction tuning is critical to large language models (LLMs) for achieving better instruction following and task adaptation capabilities but its success heavily relies on the training data quality. Many recent methods focus on improving the data quality but often overlook the compatibility of the data with the student model being finetuned. This paper introduces Selective Reflection-Tuning, a novel paradigm that synergizes a teacher LLM’s reflection and introspection for improving existing data quality with the data selection capability of the student LLM, to automatically refine existing instruction-tuning data. This teacher-student collaboration produces high-quality and student-compatible instruction-response pairs, resulting in sample-efficient instruction tuning and LLMs of superior performance. Selective Reflection-Tuning is a data augmentation and synthesis that generally improves LLM finetuning and self-improvement without collecting brand-new data. We apply our method to Alpaca and WizardLM data and achieve much stronger and top-tier 7B and 13B LLMs.
BibTeX
@inproceedings{li2024selective,
  author = {Ming Li and Lichang Chen and Jiuhai Chen and Shwai He and Jiuxiang Gu and Tianyi Zhou},
  title = {{Selective Reflection-Tuning: Student-Selected Data Recycling for LLM Instruction-Tuning}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  url = {https://openreview.net/forum?id=nvXucRIPJ0},
}
Figure