WildReward: Learning Reward Models from In-the-Wild Human Interactions

Hao Peng, Yunjia Qi, Xiaozhi Wang, Zijun Yao, Lei Hou, Juanzi Li

TL;DR

Researchers from Tsinghua University have developed WildReward, a reward model trained directly on "in-the-wild" human-LLM interactions rather than expensive, manually annotated preference pairs. By leveraging a novel pipeline to extract implicit feedback from real user logs, they created a model that rivals or outperforms much larger 70B-parameter models in accuracy, calibration, and global consistency.

What Problem Does This Paper Address?

Training Large Language Models (LLMs) to align with human preferences typically requires Reinforcement Learning from Human Feedback (RLHF). The bottleneck of this process is the Reward Model (RM), which usually needs tens of thousands of human-annotated "preference pairs" (where a human chooses which of two responses is better). This is slow, expensive, and often results in models that are biased toward superficial cues like response length.

Meanwhile, millions of people interact with LLMs daily, providing a gold mine of "in-the-wild" feedback. However, this data is messy:

  1. Feedback Sparsity: Most users don't say "good job"; they just ask the next question. Explicit positive feedback is rare (~1%).
  2. Feedback Noise: Users might give negative feedback even when a model correctly refuses a harmful request.

WildReward aims to bridge this gap by turning these messy, natural interactions into high-quality training signals.

Figure 1: Illustration of a human-LLM interaction with implicit feedback signals in the conversation. The user provides valid feedback and identifies an error.

Methods

The authors proposed a multi-stage pipeline to transform the WildChat dataset (1 million ChatGPT logs) into a curated training set called WildFB.

  1. Feedback Classification: They used a strong LLM (gpt-oss-120b) to categorize user follow-up queries into five levels of satisfaction: Explicit Rejection, Error Correction, Neutral Ambiguity, Positive Engagement, and Explicit Satisfaction.
  2. Implicit Feedback Mining: To combat sparsity, they looked for "Neutral" turns that were semantically similar to a "Positive" turn within a two-turn window, reclassifying them as positive signals.
  3. Refusal Validation: They identified cases where users gave negative feedback to a model that was actually being safe (e.g., refusing to generate hate speech) and corrected those labels.
  4. Ordinal Regression: Unlike standard RMs that use the Bradley-Terry model for pairwise ranking, WildReward uses ordinal regression. It learns to predict the probability that a response exceeds a certain quality level (1 to 4). This allows the model to assign an absolute score to a single response, rather than just comparing two.
Figure 2: Overview of the proposed pipeline for extracting human feedback from in-the-wild conversations.

Datasets

  • WildChat: The primary source of 1M real-world human-LLM conversations.
  • WildFB (New): A curated subset of 186k instances with 4-level quality labels derived from the authors' pipeline.
  • Benchmarks: The model was evaluated on RewardBench, RM-Bench, PPE, and JudgeBench, covering domains like coding, math, and safety.

Key Insights and Ideas

  • User Diversity is a Superpower: The authors found that a reward model trained on data from 10x more unique users performed significantly better, even if the total number of training samples remained the same. Diverse users provide a more robust signal of "general" human preference.
  • Global vs. Local Calibration: Traditional pairwise RMs are good at saying "A is better than B" (local), but bad at saying "A is a 7/10 and C is a 4/10" (global). Because WildReward uses ordinal regression on absolute feedback levels, it achieves much better cross-sample consistency.
  • Robustness to Length Bias: Because real users often complain when a model is wordy but wrong, WildReward is less susceptible to the "verbosity bias" that plagues models trained on paid annotators (who often prefer longer answers).

Key Results

  • Efficiency: WildReward-8B outperformed several 70B-parameter models (like Llama-3.1-Nemotron-70B) on benchmarks like RM-Bench Hard and PPE Correctness.
  • Calibration: The model achieved an Expected Calibration Error (ECE) of just 2.76%, meaning its confidence scores are highly reliable indicators of actual accuracy.
  • DPO Gains: When used to guide Online Direct Preference Optimization (DPO), WildReward led to significant improvements in downstream tasks. For example, it boosted Llama-3.1-8B-Instruct's performance on Arena Hard from 55.7% to 63.5%.

Limitations and Future Work

  • Language Scope: The current study focuses only on English and Chinese.
  • RL Training: The authors used DPO for policy alignment but did not explore more resource-intensive Reinforcement Learning (PPO) methods.
  • Dynamic Evolution: The paper suggests a future where reward models "co-evolve" with users, continuously learning from new interactions in real-time, though this was not implemented here.

For those interested in the implementation, the code and data are available at https://github.com/THU-KEG/WildReward.

Summary updated

Abstract

Reward models (RMs) are crucial for the training of large language models (LLMs), yet they typically rely on large-scale human-annotated preference pairs. With the widespread deployment of LLMs, in-the-wild interactions have emerged as a rich source of implicit reward signals. This raises the question: Can we develop reward models directly from in-the-wild interactions? In this work, we explore this possibility by adopting WildChat as an interaction source and proposing a pipeline to extract reliable human feedback, yielding 186k high-quality instances for training WildReward via ordinal regression directly on user feedback without preference pairs. Extensive experiments demonstrate that WildReward achieves comparable or even superior performance compared to conventional reward models, with improved calibration and cross-sample consistency. We also observe that WildReward benefits directly from user diversity, where more users yield stronger reward models. Finally, we apply WildReward to online DPO training and observe significant improvements across various tasks. Code and data are released at https://github.com/THU-KEG/WildReward.
BibTeX
@inproceedings{peng2026wildreward,
  author = {Hao Peng and Yunjia Qi and Xiaozhi Wang and Zijun Yao and Lei Hou and Juanzi Li},
  title = {{WildReward: Learning Reward Models from In-the-Wild Human Interactions}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.864},
}
Figure