Online Learning from Strategic Human Feedback in LLM Fine-Tuning

Shugang Hao, Lingjie Duan

TL;DR

This paper tackles the challenge of "strategic" human labelers in LLM fine-tuning, where labelers might intentionally misreport preferences to influence the model. The authors propose a novel online learning mechanism that dynamically adjusts labeler weights based on feedback accuracy, achieving a vanishing regret and outperforming existing methods. This is crucial for building more reliable and aligned LLMs by ensuring feedback is both truthful and effectively utilized.

What Problem Does This Paper Address?

Fine-tuning Large Language Models (LLMs) to align with human preferences is a critical step, often achieved through Reinforcement Learning from Human Feedback (RLHF). However, a significant challenge arises from the nature of human feedback itself. Labelers are not always perfectly rational or altruistic; they can have diverse preferences and might strategically misreport their feedback to steer the LLM towards their own biases. This "strategic feedback" can lead to a non-vanishing "regret," meaning the LLM's performance loss doesn't decrease over time, even with more data. Current RLHF practices often simply average feedback, failing to identify and leverage the most accurate labelers, thus exacerbating this problem. This paper addresses the gap in understanding and mitigating strategic human feedback in the online LLM fine-tuning process, aiming for truthful feedback and sublinear regret.

Methods

The core of the paper's contribution lies in its novel online learning mechanism. The authors frame the problem as a dynamic Bayesian game. Their approach involves three stages in each time slot of the LLM fine-tuning process:

  1. Online Preference Feedback: The system generates prompts and responses, and human labelers provide their preferences. Crucially, labelers can strategically misreport their true preferences.
  2. Online Feedback Aggregation and Policy Optimization: The system aggregates the feedback using a weighted sum. Standard LLM fine-tuning typically uses uniform weights here. The paper uses this stage to train the LLM policy.
  3. Reweighing Human Labelers: This is the novel stage. Instead of uniform weights, the system dynamically adjusts each human labeler's weight for the next time slot based on their feedback accuracy in the current time slot. Specifically, a labeler whose feedback is closer to the realized ground-truth preference receives a higher weight. This is achieved through a weighted aggregation mechanism with a step-size parameter α.

The authors also analyze two benchmark schemes:

  • Benchmark 1: Average Feedback Aggregation: This is the current standard, using uniform weights for all labelers.
  • Benchmark 2: Median Aggregation Scheme: A common approach in algorithmic game theory, which selects the median feedback.

Datasets

The paper uses a synthetic data generation approach for simulating human labelers' preferences. They mention using the Reddit TL;DR summarization dataset to draw prompts (form posts) and a supervised fine-tuning model to obtain the reference policy. The pairwise summaries (responses) are generated by this policy.

Key Insights and Ideas

  • Strategic Labelers are a Real Problem: The paper highlights that simply averaging feedback is insufficient because human labelers can and will strategically misreport preferences to influence the LLM, leading to persistent performance loss (non-vanishing regret).
  • Dynamic Weighting is Key: The core insight is that by dynamically adjusting labeler weights based on their observed accuracy, the system can incentivize truthful feedback and effectively identify the most reliable labelers over time.
  • Truthfulness and Efficiency are Achievable: The proposed mechanism is shown to be truthful (labelers are incentivized to be honest) and efficient (achieves sublinear regret), a significant improvement over existing methods.
  • The "Median" Scheme Isn't Always Truthful: Counter-intuitively, a common game-theoretic approach like the median scheme is shown to not guarantee truthful feedback in this online LLM context.

Key Results

  • Regret Analysis:
    • Benchmark 1 (Average Feedback) and Benchmark 2 (Median Aggregation) both incur a non-vanishing time-average regret of $O(T)$, meaning their performance loss does not decrease as the number of time slots ($T$) increases.
  • The proposed mechanism achieves a sublinear time-average regret of $O(\sqrt{T})$ (specifically $O(T^{1/2})$), which vanishes as $T \to \infty$. This means the LLM's performance loss can be made arbitrarily small with enough training time.
  • Simulation Results:
    • Figure 1 demonstrates how the proposed mechanism dynamically adjusts labeler weights, giving higher weights to more accurate labelers over time.
  • Figure 2 visually confirms that the proposed mechanism significantly reduces time-average regret compared to the two benchmarks, with the regret approaching zero as $T$ increases.

Limitations and Future Work

  • Synthetic Data: The experiments rely on synthetic data for human labeler preferences. While this allows for controlled analysis, future work could explore real-world human feedback data.
  • Assumptions: The mechanism assumes a Bernoulli belief for human labelers' preferences.
  • Step-size Parameter: The choice of the step-size parameter α influences the convergence rate. While an optimal choice is derived for theoretical bounds, practical tuning might be necessary.

The authors suggest that their work lays the foundation for designing truthful and efficient online learning mechanisms for LLM fine-tuning, paving the way for more robust and aligned AI systems.

Summary updated

Abstract

Reinforcement learning from human feedback (RLHF) has become an essential step in fine-tuning large language models (LLMs) to align them with human preferences. However, human labelers are selfish and have diverse preferences. They may strategically misreport their online feedback to influence the system's aggregation towards their own preferences. Current practice simply averages labelers' feedback per time and fails to identify the most accurate human labeler, leading to linear regret $\mathcal{O}(T)$ for $T$ time slots. To our best knowledge, we are the first to study online learning mechanisms against strategic human labelers in the LLM fine-tuning process. We formulate a new dynamic Bayesian game and dynamically adjust human labelers' weights in the preference aggregation, ensuring their truthful feedback and sublinear regret $\mathcal{O}(T^{1/2})$. Simulation results demonstrate our mechanism's great advantages over the existing benchmark schemes.
BibTeX
@inproceedings{hao2025online,
  author = {Shugang Hao and Lingjie Duan},
  title = {{Online Learning from Strategic Human Feedback in LLM Fine-Tuning}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2025},
  doi = {10.1109/icassp49660.2025.10887891},
}
Figure