Dynamic Noise Preference Optimization for LLM Self-Improvement via Synthetic Data

Haoyan Yang, Ting Hua, Shangqian Gao, Binfeng Xu, Zheng Tang, Jie Xu, Hongxia Jin, Vijay SrinivasanSubmission to ICLR 2025 · Rejected

TL;DR

Self-play fine-tuning methods like SPIN quickly hit a wall because human ground-truth data often contains errors or unhelpful refusals, and widening probability margins cause training gradients to vanish. Dynamic Noise Preference Optimization (DNPO) resolves this by dynamically relabeling preference pairs using an LLM evaluator and injecting trainable noise into the reference policy to shrink optimization margins. This revives gradient flow and enables consistent, multi-round self-improvement without human intervention.

What Problem Does This Paper Address?

Post-training alignment usually relies heavily on human-annotated preference data, which is expensive and non-scalable. Recent self-play techniques (such as SPIN) attempt autonomous self-improvement by iteratively training a model to distinguish human "ground truth" (positive) from its own synthetic outputs (negative).

However, these approaches typically suffer from severe performance stagnation after a single iteration due to two main flaws:

  1. The "Human is Always Better" Myth: In practice, human data is noisy. The authors found that ~30% of the time, the model's self-generated output was actually equal or superior to the human response (e.g., when human annotators gave generic refusals like "I am incapable of purchasing clothes" to a creative prompt). Rigidly treating human data as positive introduces roughly 30% label noise into preference training.
  2. Vanishing Gradients from Margin Saturation: After the first iteration, the model's predicted probability margin between positive and negative samples becomes large and rigid. The resulting preference loss produces negligible gradients, trapping the model in a suboptimal state with minimal variation across rounds.

Methods

DNPO tackles these failure modes with two complementary mechanisms:

[Iter k Model] ──> Generate Answers 
       │
       ├──> Dynamic Sample Labeling (DSL): 
       │      Judge LLM scores (Synthetic vs. Human) ──> Higher score = Positive, Lower = Negative
       │
       └──> Noise Preference Optimization (NPO):
              Inject trainable Gaussian noise into negative logits of the reference model 
              Min-Max formulation: Noise shrinks the margin; Model parameters widen it
  • Dynamic Sample Labeling (DSL): Instead of assuming human annotations are always positive, an external evaluator (GPT-4o-mini) scores both the human reference and the model-generated output. Whichever achieves the higher score is assigned as $y^+$, and the other becomes $y^-$.
  • Noise Preference Optimization (NPO): To prevent gradient starvation, the authors frame the objective as a min-max game. They inject trainable zero-mean Gaussian noise into the logits of negative samples under the reference policy ($p_{\theta_t}$). A lightweight linear projection ($\theta_\sigma = [W_\sigma, b_\sigma]$) predicts the noise variance from the reference model's last hidden state. The noise parameters update to minimize the margin between positive and negative pairs (making the distinction harder), while the language model updates to maximize the margin. Both components are updated simultaneously in a single step with a variance penalty term $\alpha$.

Datasets

  • Training: A 20k subset of UltraChat-200k, using Mistral-7B fine-tuned into Zephyr-7B-SFT as the base model.
  • Evaluation Benchmarks: Evaluated using zero-shot / few-shot normalized accuracy on standard benchmarks: ARC (25-shot), TruthfulQA (0-shot), Winogrande (5-shot), GSM8K (5-shot), HellaSwag (10-shot), and MMLU (5-shot).
  • Generation Quality Evaluation: Pairwise and absolute scoring using frontier LLMs as judges (GPT-4o-mini, GPT-4o, and Claude 3.5-Haiku), alongside classical lexical and semantic metrics (BLEU, ROUGE-L, and Sentence-BERT similarity). No new dataset benchmarks were introduced.

Key Insights and Ideas

  • Synthetic Data Can Clean Human Datasets: In conversational and instruction tasks, LLMs often follow instructions more diligently than hurried human crowd-workers. Recognizing this flipped dynamic and dynamically relabeling samples removes harmful supervision.
  • Controlled Noise Overcomes Self-Play Stagnation: Random noise in high-dimensional token spaces fails to counteract margin collapse. However, parameterizing and training the perturbation variance directly on reference hidden states selectively blurs the contrastive boundary, forcing the model to generate meaningful gradient updates in later iterations.
  • Iterative Balance: Analysis of loss dynamics reveals an adversarial balance: noise loss decreases as it learns where the model is overly confident, keeping training dynamic across epochs rather than plateauing.

Key Results

  • Sustained Benchmark Gains: While SPIN peaked at Iteration 0 (0.596) and deteriorated to 0.586 by Iteration 3, DNPO steadily climbed from 0.587 (SFT) to 0.612 at Iteration 3—a +2.6% overall gain over SPIN.
  • TruthfulQA & ARC Outperformance: DNPO achieved 0.417 on TruthfulQA (vs. 0.383 for SPIN and 0.340 for SFT, a +7.7% boost) and 0.737 on ARC (vs. 0.703 for SPIN).
  • Head-to-Head Generation Win Rates: In direct GPT-4o-mini evaluations against SPIN outputs, DNPO secured a 57.51% win rate vs. 28.07% loss at Iteration 3 (a 29.4% gap). Similar win rates held under Claude 3.5-Haiku (57.85% win rate) and GPT-4o (58.2% win rate).
  • Ablation Evidence: Both DSL and NPO independently outperformed baseline SPIN across all iterations, with the combined DNPO yielding the highest performance. Trainable noise consistently surpassed fixed Gaussian perturbations ($N(0, 0.5)$).

Limitations and Future Work

  • Dependence on an External Judge Model: DSL relies on proprietary models (GPT-4o-mini) to score pairs during data curation, incurring API costs and inheriting the judge model's latent biases. Autonomous self-rewarding language models were not explored in this pipeline.
  • Scale and Scope: Experiments were confined to 7B-parameter models (Mistral/Zephyr) and up to three self-play iterations. Whether these dynamics hold for larger models (70B+) or more complex multi-step reasoning domains remains unverified.

Summary updated

Abstract

Although LLMs have achieved significant success, their reliance on large volumes of human-annotated data has limited their potential for further scaling. In this situation, utilizing self-generated synthetic data has become crucial for fine-tuning LLMs without extensive human annotation. However, current methods often fail to ensure consistent improvements across iterations, with performance stagnating after only minimal updates. To overcome these challenges, we introduce Dynamic Noise Preference Optimization (DNPO). DNPO employs a dynamic sample labeling mechanism to construct preference pairs for training and introduces controlled, trainable noise into the preference optimization process. Our approach effectively prevents stagnation and enables continuous improvement. In experiments with Zephyr-7B, DNPO consistently outperforms existing methods, showing an average performance boost of 2.6\% across multiple benchmarks. Additionally, DNPO shows a significant improvement in model-generated data quality, with a 29.4\% win-loss rate gap compared to the baseline in GPT-4 evaluations. This highlights its effectiveness in enhancing model performance through iterative refinement.
BibTeX
@inproceedings{yang2026dynamic,
  author = {Haoyan Yang and Ting Hua and Shangqian Gao and Binfeng Xu and Zheng Tang and Jie Xu and Hongxia Jin and Vijay Srinivasan},
  title = {{Dynamic Noise Preference Optimization for LLM Self-Improvement via Synthetic Data}},
  booktitle = {RSI @ ICLR},
  year = {2026},
  url = {https://openreview.net/forum?id=nT1gTSAf07},
}
Figure