Dynamic Noise Preference Optimization for LLM Self-Improvement via Synthetic Data

Haoyan Yang, Ting Hua, Shangqian Gao, Binfeng Xu, Zheng Tang, Jie Xu, Hongxia Jin, Vijay Srinivasan

Training state-of-the-art Large Language Models (LLMs) has traditionally relied on massive quantities of high-quality, human-annotated data. However, as human data becomes increasingly scarce and expensive, the research community is hitting a "data wall." While self-improvement through synthetic data generation is a promising path forward, existing methods often suffer from two critical issues: the "human-is-always-better" fallacy and model stagnation.

This paper introduces Dynamic Noise Preference Optimization (DNPO), a framework designed to overcome these hurdles by dynamically re-labeling preference pairs and injecting trainable noise to sustain the learning signal across multiple iterations.

The DNPO Framework

DNPO moves away from the rigid assumption that human-annotated data is inherently superior to model-generated data. It consists of two primary components: Dynamic Sample Labeling (DSL) and Noise Preference Optimization (NPO).

Overview of the DNPO framework, illustrating the iterative loop between model generation, Dynamic Sample Labeling (DSL), and Noise Preference Optimization (NPO).

Dynamic Sample Labeling (DSL)

Standard alignment methods like DPO or SPIN typically treat human data as the "positive" sample and synthetic data as the "negative." However, empirical analysis shows that models can generate better responses than human ground truths in approximately 30% of cases. DSL addresses this by using a high-capacity "Evaluation Model" (e.g., GPT-4o-mini) to score both human and synthetic responses. The higher-scoring response is designated as the positive sample, ensuring the model always learns from the highest-quality data available.

Noise Preference Optimization (NPO)

In iterative self-improvement, the model's output distribution often converges toward the target distribution so closely that the gradient signal vanishes, leading to stagnation. NPO introduces a min-max optimization game using trainable noise:

  • The Noise Generator is trained to minimize the margin between positive and negative samples, making them harder to distinguish.
  • The LLM is trained to maximize this margin.

By constantly shrinking the margin, NPO forces the model to identify more nuanced features, preventing the learning process from flatlining.

Experimental Results

The researchers evaluated DNPO using a Mistral-7B base model fine-tuned on UltraChat-200k. Performance was measured across six standard benchmarks, including ARC, TruthfulQA, and MMLU.

Log-probability distributions for positive, negative, and generated samples across four iterations, showing how the model's generated data distribution shifts over time.

Key Findings

  • Consistent Performance Gains: Unlike baseline methods like SPIN, which often stagnate after a single iteration, DNPO showed steady improvement over three iterations. It achieved an average benchmark score of 0.612, a 2.6% improvement over SPIN.
  • Factuality and Reasoning: The model demonstrated a significant 7.7% improvement on TruthfulQA compared to the SFT baseline, suggesting that dynamic labeling is particularly effective for factual accuracy.
  • Qualitative Superiority: Head-to-head evaluations using GPT-4 as a judge revealed a 29.4% win-loss rate gap in favor of DNPO-generated data over the baseline, confirming that the model produces higher-quality text.

Conclusion

DNPO provides a robust solution to the data scarcity problem by enabling effective iterative self-improvement. By abandoning the assumption of human data superiority and using trainable noise to maintain a strong learning signal, DNPO allows LLMs to continue evolving using their own synthetic outputs. Future work will explore scaling this framework to larger models (70B+) and specialized domains like mathematical reasoning.

Summary updated

BibTeX
@inproceedings{yang2026dynamic,
  author = {Haoyan Yang and Ting Hua and Shangqian Gao and Binfeng Xu and Zheng Tang and Jie Xu and Hongxia Jin and Vijay Srinivasan},
  title = {{Dynamic Noise Preference Optimization for LLM Self-Improvement via Synthetic Data}},
  booktitle = {RSI @ ICLR},
  year = {2026},
  url = {https://openreview.net/forum?id=nT1gTSAf07},
}
Figure