Beyond Sample-Level Feedback: Using Reference-Level Feedback to Guide Data Synthesis
Lists
Only you see this
Abstract
High-quality instruction-tuning data is crucial for developing Large Language Models (LLMs) that can effectively navigate real-world tasks and follow human instructions. While synthetic data generation offers a scalable approach for creating such datasets, it imposes a quality ceiling where models trained on the data cannot outperform the LLM generating it. To overcome this limitation, we introduce Reference-Level Feedback, a paradigm that extracts desirable characteristics from carefully curated reference samples to guide the synthesis of higher-quality instruction-response pairs. Using this approach, we synthesize REFED, a dataset of 10K instruction-response pairs. Fine-tuning Llama-3.1-8B-Instruct and Mistral-7B-Instruct on REFED demonstrate state-of-the-art performance among similarly sized models, notably reaching a 43.96\% length-controlled win-rate on AlpacaEval 2.0. Extensive experiments demonstrate that Reference-Level Feedback consistently outperforms traditional sample-level feedback methods, generalizes across model architectures, and produces high-quality and diverse data at low cost.
BibTeX
@misc{mehri2025beyond,
author = {Shuhaib Mehri and Xiusi Chen and Heng Ji and Dilek Hakkani-Tür},
title = {{Beyond Sample-Level Feedback: Using Reference-Level Feedback to Guide Data Synthesis}},
year = {2025},
eprint = {2502.04511},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2502.04511},
}Loading related content...