Beyond Sample-Level Feedback: Using Reference-Level Feedback to Guide Data Synthesis

Shuhaib Mehri, Xiusi Chen, Heng Ji, Dilek Hakkani-Tür

Abstract

LLMs demonstrate remarkable capabilities in following natural language instructions, largely due to instruction-tuning on high-quality datasets. While synthetic data generation has emerged as a scalable approach for creating such datasets, maintaining consistent quality standards remains challenging. Recent approaches incorporate feedback to improve data quality, but typically operate at the sample level, generating and applying feedback for each response individually. In this work, we propose Reference-Level Feedback, a novel methodology that instead collects feedback based on high-quality reference samples from carefully curated seed data. We use this feedback to capture rich signals of desirable characteristics and propagate it throughout the data synthesis process. We present REFED, a dataset of 10K instruction-response pairs synthesized using such feedback. We demonstrate the effectiveness of our approach by showing that Llama-3.1-8B-Instruct finetuned on REFED achieves state-of-the-art performance among similar-sized SFT-based models on AlpacaEval 2.0 and strong results on Arena-Hard. Through extensive experiments, we show that our approach consistently outperforms traditional sample-level feedback methods with significantly fewer feedback collections and improves performance across different model architectures.

Introduction: The Quality Ceiling in Synthetic Data

Instruction tuning is critical for aligning Large Language Models (LLMs) with human intent. However, the reliance on human-curated data limits scalability. Synthetic data generation offers a solution but often hits a "quality ceiling," where the student model's performance is capped by the teacher model's average output quality.

Traditional methods attempt to break this ceiling using sample-level feedback, where a stronger model critiques every individual generated response. While effective, this approach is computationally expensive—scaling linearly $O(n)$ with dataset size—and often misses the underlying principles that define high-quality data.

Comparison between traditional sample-level feedback and the proposed reference-level feedback paradigm.

The REFED Paradigm: Reference-Level Feedback

To address these limitations, researchers at UIUC introduced Reference-Level Feedback (REFED). Instead of critiquing every new sample, REFED extracts high-level quality principles from a small set of "gold-standard" human-curated samples (e.g., the LIMA dataset). These principles are then used as a blueprint to guide the synthesis of thousands of new examples. This shifts the feedback complexity from $O(n)$ to $O(1)$ relative to the synthesized data size, as feedback is collected once per reference and applied to many generated samples.

Methodology: A Three-Step Synthesis Pipeline

The REFED framework operates through a structured three-step process:

  1. Reference-Level Feedback Collection: High-quality reference samples are analyzed to extract "Instruction Feedback" (identifying subject areas and required skills) and "Response Feedback" (identifying specific strengths like tone, clarity, and depth).
  2. Instruction Synthesis: The LLM uses the instruction feedback as a guideline to generate multiple new, diverse instructions that mirror the complexity of the original reference.
  3. Response Synthesis & Refinement: Initial responses are generated for the new instructions. These are then refined using the original reference-level response feedback, ensuring that the "DNA" of the high-quality seed data is propagated into the synthetic dataset.
The three-step REFED pipeline: Feedback Collection, Instruction Synthesis, and Response Synthesis and Refinement.

Finally, an LLM-as-a-judge filtering step ensures only samples with significant quality improvements are retained in the final 10K-sample REFED dataset.

Experimental Results and Performance

The effectiveness of REFED was evaluated by fine-tuning a Llama-3.1-8B-Instruct model. The results demonstrate that reference-level guidance can push models beyond the typical synthetic data ceiling:

  • State-of-the-Art Benchmarks: The REFED-tuned model achieved a 43.96% length-controlled win rate on AlpacaEval 2.0, the highest among similarly sized models. This is nearly double the performance of the base Llama-3.1-8B-Instruct model (22.90%).
  • Cost Efficiency: By leveraging GPT-4o mini for synthesis and refinement, the entire 10K dataset was produced for less than $20, making it significantly more accessible than previous feedback-heavy methods.
  • Data Diversity: Metrics for syntactic diversity and distinct-n grams show that REFED produces more diverse data than popular alternatives like Evol-Instruct, despite using structured feedback templates.

Discussion and Future Work

REFED demonstrates that explicitly extracting and propagating "desirable characteristics" from reference data is a powerful and efficient way to scale instruction tuning. While the method still depends on a strong teacher model and the quality of the initial seed data, it provides a clear path toward breaking the performance plateau of synthetic datasets. Future research will focus on optimizing seed data selection and exploring the limits of data scaling from a fixed reference set.

Summary updated

Abstract

LLMs demonstrate remarkable capabilities in following natural language instructions, largely due to instruction-tuning on high-quality datasets. While synthetic data generation has emerged as a scalable approach for creating such datasets, maintaining consistent quality standards remains challenging. Recent approaches incorporate feedback to improve data quality, but typically operate at the sample level, generating and applying feedback for each response individually. In this work, we propose Reference-Level Feedback, a novel methodology that instead collects feedback based on high-quality reference samples from carefully curated seed data. We use this feedback to capture rich signals of desirable characteristics and propagate it throughout the data synthesis process. We present REFED, a dataset of 10K instruction-response pairs synthesized using such feedback. We demonstrate the effectiveness of our approach by showing that Llama-3.1-8B-Instruct finetuned on REFED achieves state-of-the-art performance among similar-sized SFT-based models on AlpacaEval 2.0 and strong results on Arena-Hard. Through extensive experiments, we show that our approach consistently outperforms traditional sample-level feedback methods with significantly fewer feedback collections and improves performance across different model architectures.
BibTeX
@misc{mehri2025beyond,
  author = {Shuhaib Mehri and Xiusi Chen and Heng Ji and Dilek Hakkani-Tür},
  title = {{Beyond Sample-Level Feedback: Using Reference-Level Feedback to Guide Data Synthesis}},
  year = {2025},
  eprint = {2502.04511},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2502.04511},
}
Figure