Improving Preference Alignment of LLM with Inference-Free Self-Refinement

Fukun Ma, Kaibin Tian, Jieting Xue, Xiaoyi Wang, Ye Ma, Quan Chen, Peng Jiang, Lijie WenEMNLP 2025

TL;DR

This paper introduces "Inference-Free Self-Refinement" (IFSR), a novel approach to improve how Large Language Models (LLMs) align with human preferences. IFSR leverages a fixed "refining instruction" and a "pseudo reference response" to guide LLMs during training, leading to significant performance gains without requiring complex inference-time self-correction.

What Problem Does This Paper Address?

Large Language Models (LLMs) have become incredibly capable, but ensuring their outputs are helpful, harmless, and aligned with human intent is a major challenge. While techniques like Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO) have made strides, they often rely on complex training loops or struggle with distributional gaps between model generations and desired preferences. The paper identifies a phenomenon where LLMs can improve their outputs simply by being prompted with specific instructions, even without explicit feedback or initial responses. This "Inference-Free Self-Refinement" (IFSR) is observed to be an inherent capability that can be leveraged for better preference alignment.

Methods

The core of IFSR lies in two components:

  • Refining Instruction: A fixed, simple instruction (e.g., "please generate a better response") that guides the LLM's output distribution from a preference-semantic perspective. This instruction is incorporated during training.
  • Pseudo Reference Response: A generated response derived from paired preference data, serving as a demonstration to guide the LLM's output. This helps mitigate off-policy distributional bias and enhances token-level preference learning.

IFSR is integrated into existing LLM alignment pipelines (like instruction fine-tuning and preference optimization) by augmenting the training data with these components. This allows the model to learn to refine its outputs implicitly during training, rather than relying solely on explicit self-correction during inference.

Datasets

The paper uses four public datasets for training and evaluation:

  • OpenAssistant Conversations Dataset (OASST)
  • UltraFeedback Binarized Dataset (UltraBin)
  • Stanford Human Preferences Dataset (SHP)
  • Anthropic Helpful and Harmless Dataset (HH)

Key Insights and Ideas

  • Inherent Capability: IFSR is not a new capability that needs to be trained from scratch; it's an emergent property of pretrained LLMs that can be unlocked and amplified.
  • Dual Mechanism: IFSR works by combining semantic relevance constraints (from the refining instruction) with demonstration-based regularization (from the pseudo reference response).
  • Training is Key: While IFSR can be observed during inference, its true power is unleashed when integrated into the training process, leading to more substantial and consistent improvements.
  • Orthogonal Improvement: IFSR can be applied on top of existing preference optimization methods (like DPO, KTO, CPO, ORPO, SimPO) to boost their performance, suggesting it addresses a fundamental limitation.
  • Focus on Relevant Tokens: Attention analysis shows that IFSR helps models focus more on tokens that are crucial for preference alignment, leading to more nuanced and contextually appropriate responses.

Key Results

Experiments show that IFSR consistently improves preference alignment across various datasets and base models.

  • Significant Gains: IFSR achieved average improvements of 24.97% in reward model scores and 8.44 percentage points in GPT-4 win rates for the Pythia-2.8B model, and 20.99% and 7.32 percentage points respectively for the Llama3-8B model.
  • Broad Applicability: IFSR enhanced performance by an average of 12.26% when applied with different preference optimization methods (KTO, CPO, ORPO, SimPO), demonstrating its generalizability.
  • Cross-Domain Robustness: IFSR showed consistent performance enhancements even when trained on one dataset and tested on another, indicating robust generalization.

Limitations and Future Work

  • Configuration Sensitivity: The effectiveness of IFSR can be sensitive to hyperparameters, requiring careful tuning.
  • Evaluation Metrics: Current metrics might not fully capture human preferences, potentially favoring longer responses.
  • Computational Cost: While IFSR improves performance, it does add computational costs equivalent to baseline optimization methods, which could impact broader adoption.
  • Future Work: The authors suggest expanding experimental settings to include more base models, optimization methods, and datasets to further explore IFSR's phenomena. Investigating the impact of different refining instructions and further analyzing the underlying mechanisms are also promising directions.

Summary updated

BibTeX
@inproceedings{ma2025improving,
  author = {Fukun Ma and Kaibin Tian and Jieting Xue and Xiaoyi Wang and Ye Ma and Quan Chen and Peng Jiang and Lijie Wen},
  title = {{Improving Preference Alignment of LLM with Inference-Free Self-Refinement}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure