Efficiently Selecting Response Generation Strategies for Synthetic Data Construction by Self-Aligned Perplexity

Xuan Ren, Qi Chen, Lingqiao Liu2025 · DOI

Abstract

Fine-tuning large language models (LLMs) typically relies on producing large sets of input-output pairs. Yet for a given question, there can be many valid outputs. In practice, these outputs are often derived by distilling knowledge from teacher models, and they can vary depending on the specific teacher model or prompting strategy employed. Recent findings show that how these training outputs are generated can significantly affect the performance of the fine-tuned model, raising an important question: how do we pick the best data generation method from among numerous possibilities? Rather than exhaustively training and evaluating on each candidate, this paper proposes a scalable approximate method that assesses a small subset of generated data to estimate its suitability for a specific target LLM. Our central idea is that effective outputs should be familiar to the target LLM. While previous work measures familiarity with perplexity, we find that perplexity might be suboptimal in characterizing "familiarity" through empirical analyses and practical observations. To address this, we introduce self-aligned perplexity, a novel metric capturing how closely candidate outputs adhere to the target LLM's own style and reasoning patterns. In this way, we can identify the most effective generation strategy on a small sample, then apply it to produce the complete training set. We demonstrate that training on data generated by the chosen method yields significant improvements across diverse reasoning-focused benchmarks, particularly in cases where different candidate methods lead to highly divergent training outcomes.

Introduction

Fine-tuning a "student" LLM often involves using a "teacher" model (e.g., GPT-4) to generate synthetic training examples. However, the generation strategy—whether providing direct answers, step-by-step rationales, or redundant explanations—significantly impacts the student's final performance. Because the optimal strategy varies by task and model, the traditional approach of exhaustive trial-and-error training is computationally prohibitive. This research introduces a scalable method to "preview" data effectiveness using only a tiny fraction of the total dataset.

Methodology: Self-Aligned Perplexity (SPPL)

The proposed metric, Self-Aligned Perplexity (SPPL), is based on the "Familiarity Hypothesis," which posits that models learn most effectively from data that aligns with their inherent linguistic and reasoning patterns.

Standard perplexity (PPL) is often used to measure familiarity but is prone to "hacking"; for instance, a model may assign low PPL to wordy or redundant responses simply due to fluency, even if the style is a poor fit for reasoning. SPPL addresses this by "anchoring" the measurement to the model's own voice:

  1. Style Pool Creation: The target model generates zero-shot solutions for a few problems to establish a "style pool."
  2. In-Context Anchoring: When evaluating a candidate response, two correct examples from the style pool are prepended as in-context examples.
  3. Evaluation: This forces the model to evaluate candidate responses through the lens of its own successful reasoning style, increasing the score if the candidate deviates from the model's preferred patterns.

Experimental Results

The method was evaluated across 17 diverse reasoning tasks and 6 planning tasks. Results indicate that SPPL can accurately rank generation strategies using as few as 50 samples.

Metric Performance and Correlation

SPPL achieved a Weighted Spearman correlation of 0.324, significantly outperforming standard perplexity (0.301) and other metrics such as IFD (0.191) and CAR (0.279). This demonstrates that self-alignment provides a more robust signal for data quality than raw linguistic fluency.

Figure 2: Comparison of SPPL (Ours) against standard perplexity and other selection metrics across different models.

The Improvement Ratio Threshold

Strategy selection is most critical when a model has significant "headroom" to grow. For tasks where the model already exhibits high zero-shot accuracy, the choice of strategy has a negligible impact. However, in difficult tasks characterized by a high "improvement ratio" (the ratio of potential performance gain), the choice of strategy becomes vital. On these high-variance tasks, training on data selected by SPPL improved accuracy by 4.54% over the average of other strategies.

Figure 1: Standard Deviation of performance vs. Improvement Ratio, highlighting that strategy selection matters most in high-growth scenarios.

Conclusion and Limitations

SPPL provides a robust, scalable alternative to exhaustive training for synthetic data strategy selection. While highly effective for 7B-class models, future work is required to validate the approach on larger 70B+ models and newer "reasoning" architectures. Additionally, the method depends on the model's ability to generate at least some correct zero-shot answers to form a style pool; tasks where the model is completely incapable may require alternative anchoring strategies.

Summary updated

Abstract

Fine-tuning large language models (LLMs) typically relies on producing large sets of inputoutput pairs.Yet for a given question, there can be many valid outputs.In practice, these outputs are often derived by distilling knowledge from teacher models, and they can vary depending on the specific teacher model or prompting strategy employed.Recent findings show that how these training outputs are generated can significantly affect the performance of the fine-tuned model, raising an important question: how do we pick the best data generation method from among numerous possibilities?Rather than exhaustively training and evaluating on each candidate, this paper proposes a scalable approximate method that assesses a small subset of generated data to estimate its suitability for a specific target LLM.Our central idea is that effective outputs should be familiar to the target LLM.While previous work measures familiarity with perplexity, we find that perplexity might be suboptimal in characterizing "familiarity" through empirical analyses and practical observations.To address this, we introduce self-aligned perplexity, a novel metric capturing how closely candidate outputs adhere to the target LLM's own style and reasoning patterns.In this way, we can identify the most effective generation strategy on a small sample, then apply it to produce the complete training set.We demonstrate that training on data generated by the chosen method yields significant improvements across diverse reasoning-focused benchmarks, particularly in cases where different candidate methods lead to highly divergent training outcomes.Our implementation is publicly available at
BibTeX
@misc{ren2025efficiently,
  author = {Xuan Ren and Qi Chen and Lingqiao Liu},
  title = {{Efficiently Selecting Response Generation Strategies for Synthetic Data Construction by Self-Aligned Perplexity}},
  year = {2025},
  doi = {10.18653/v1/2025.findings-emnlp.621},
}

Loading related content...

Figure