Editing Reward Hacks: Human Minimal Edits as Training Data for Dialogue Reward Models

A dialogue model can be trained with reinforcement learning (RL): the model writes replies, and a second model scores them. When no single "correct" reply exists, which is the normal case in conversation, that score has to come from a reference-free reward model. This is a learned scorer that reads the conversation and a candidate reply and outputs a quality number without comparing against a human-written answer. These scorers are known to be gameable. They reward replies that repeat the context, adopt authoritative speaker tags, or run long. A policy optimized against one will find those shortcuts. We propose to close this loop cheaply. Between rounds of RL, we collect the replies where the reward model and a cheap disagreement signal suggest the policy is exploiting a shortcut. Humans then minimally edit each of those replies into an acceptable one, and the reward model is retrained on the resulting pairs of hacked reply and edited reply.

The bet is about what a human minute buys. A rating says that a reply is bad, but not why. A minimal edit produces two replies that differ almost only in the exploited feature, so the reward model is shown the precise direction it was wrongly crediting. Editing an imperfect draft is also much faster than writing from scratch: model-assisted refinement cut 3D annotation time by over an order of magnitude in The Dresden in vivo OCT dataset for automatic middle ear segmentation. Because humans work asynchronously between RL rounds rather than chatting live, the loop adds no human latency to training. Because the reward model stays small, it avoids the cost of calling a large judge on every rollout.

Core Claim

Our claim is that, per unit of human time, minimal-edit contrastive pairs drawn from the policy's own reward-hacking outputs repair a reference-free dialogue reward model more than three alternatives: scalar ratings, synthetic adversarial perturbations, or supervised fine-tuning on the same edits. A repaired reward model, in turn, lets RL keep improving human-judged conversation quality past the point where a static reward model's score and human judgment diverge. The specific novelty is in two choices. The humans edit rather than rate. And the edited material is the policy's discovered exploits rather than a fixed attack list. The second choice matters because the policy may invent shortcuts that no one anticipated.

Where This Sits

The motivating evidence comes from Measuring the Robustness of Reference-Free Dialogue Evaluation Systems. That paper tests DialogRPT, UniEval, and prompted LLM judges against four attack classes and finds that high human correlation coexists with fragility. UniEval, for instance, is fooled by context repetition 76% of the time on DailyDialog. The paper is purely diagnostic and never trains a policy against these scorers. Evaluating Language Models in Realistic Conversational Contexts adds a "verbosity trap," in which LLM judges favor long, formulaic replies. It also builds a Mixture-of-Judges evaluator trained on expert ratings, but uses it only offline and relies on fully hand-scripted dialogues. We borrow its evaluator as an independent held-out judge. We do not use it as the training reward.

Two prior works use humans to shape rewards or policies. Nano trains a critic from 1–5 human ratings in an active loop for single-turn controllable generation. It also shows that naive negative feedback damages fluency unless a complementary loss is added. We differ in collecting edits instead of ratings, in working on multi-turn dialogue, and in targeting exploits the policy discovered. Dialogue Learning With Human-in-the-Loop had humans converse live and found that predicting textual teacher corrections outperformed reward-only updates (0.441 vs 0.375 on MTurk). That result supports our preference for rich corrections over scalars. We avoid its live-interaction cost by making the human step asynchronous.

The remaining works define the reward-learning and compute context. Fantastic Rewards and How to Tame Them and Dialogue Evaluation with Offline Reinforcement Learning learn dialogue rewards from static corpora or benchmark scores on MultiWOZ, so any metric flaws pass directly into the reward. Teacher Forcing Recovers Reward Functions for Text Generation extracts a cheap per-token reward from the model's own outputs, but that reward cannot be corrected from outside the model. The RL machinery comes from RL4LMs / NLPO, whose DailyDialog task in the GRUE benchmark is our main testbed. That paper also documents reward hacking and generation-in-the-loop cost. QE-EBM warns about compute: updating the scorer jointly with the policy raised step time from 0.2s to 17.2s in translation. For this reason we update the reward model only between rounds, never inside the RL step.

Research Program

The loop runs in rounds. In each round, a small policy is trained with NLPO for a fixed number of steps against the current reward model. From the round's rollouts, we flag candidate exploits using two cheap signals. The first is disagreement between the reward model and a small ensemble of alternative scorers. The second is large reward gain paired with low lexical or semantic change from generic templates. A capped number of flagged replies, together with an equal number of random replies as a control, are sent to annotators. Annotators are asked to make the smallest edit that turns the reply into a good one for this context, or to mark it acceptable as is.

Each edit $e$ of a hacked reply $h$ in context $c$ becomes a preference pair. The reward model $r_\theta$ is trained with a margin loss on the pair together with its original data:

$$\mathcal{L}(\theta) = -\log \sigma\big(r_\theta(c, e) - r_\theta(c, h) - m\big),$$

Here $\sigma$ is the logistic function and $m$ is a small margin. Following Nano's finding, we also keep a standard loss on unedited acceptable replies, so that the model does not learn to penalize everything the policy produces.

Beyond the headline comparison, we will analyze three questions. The first is which shortcut types the policy discovers over rounds, and whether any fall outside the four attack classes of the robustness study; verbosity is our prior candidate. The second is how edit distance and annotator time change as the reward model improves. The third is whether annotators fall into automation bias, which we measure by planting known-bad replies with no surface shortcut into the edit queue. We will report every cost in two currencies: GPU-hours, split into rollout, reward scoring, and reward retraining, and annotator-minutes.

Baselines

The naive reference point is supervised fine-tuning of the policy on DailyDialog, together with NLPO against a fixed reward model (UniEval, and separately a fine-tuned DialogRPT-style scorer) with no human loop. This is the setup whose reward-versus-human divergence we expect to see. The strongest published RL method on this task is NLPO from RL4LMs, which reported more stability than PPO on GRUE. However, no reported result found on Lacuna for NLPO's DailyDialog human-quality numbers, so we rerun it ourselves with the same policy size, rollout budget, and hyperparameter search as our method. For reward robustness, the published reference points are UniEval's 76% context-repetition attack success on DailyDialog and PE-GPT4's Kendall's τ of 0.562 with human judgment on TopicalChat, both from Measuring the Robustness of Reference-Free Dialogue Evaluation Systems. Beating them means lowering attack success well below 76% without dropping correlation below that of the unrepaired starting reward model.

The most serious competitor is synthetic-perturbation augmentation. We retrain the same reward model on automatically generated negatives from the four known attack classes, with a generation budget matched to the number of our human pairs. A second competitor is rating-based repair in the style of Nano. Annotators receive the same flagged replies and the same total minutes, but give 1–5 ratings instead of edits.

Two ablations isolate our ingredients. In the first, the edits are used only to fine-tune the policy directly, not to retrain the reward model; this isolates whether the reward model is the right place for the human signal. In the second, edits go to random rollouts rather than flagged ones; this isolates exploit selection. We will also use the Mixture-of-Judges from UPHELD as a training reward. It is expensive per call and is included as a compute-heavy upper reference. Our method counts as winning if, at matched annotator-minutes and within a stated GPU-hour budget, it yields higher final human-judged quality than every baseline. It must also show no late-training decline in human quality while the reward score still rises.

Why not just…?

Why not just augment the reward model with synthetic adversarial examples? This is the real threat. The four attack classes are known and cheap to generate, and if the policy only ever discovers those four shortcuts, synthetic augmentation may match our method at zero human cost. The experiment settles it in two ways. First, we hold out one attack class, speaker tags, from the synthetic set and check whether each method's final reward model resists it. Second, we catalog the exploits the policy actually discovers. If every discovered exploit falls within the known classes and synthetic augmentation ties, we will report that human editing is unnecessary for this regime.

Why not just collect ratings instead of edits? Ratings are faster per item. But in Dialogue Learning With Human-in-the-Loop, textual corrections beat scalar feedback, and Nano needed a special loss to stop negative ratings from harming fluency. Our matched-minutes comparison measures whether an edit's pinpointed contrast outweighs the higher throughput of ratings. If ratings win at equal time, the core claim fails.

Why not just fine-tune the policy on the edited replies and skip RL? Edited replies are high-quality in-context targets, and supervised training on them avoids rollout cost entirely. The weakness is scale. A few thousand edits make a small supervised set, whereas a repaired reward model can score unlimited rollouts. The edits-to-policy-only ablation tests this directly. If it matches the full loop at lower GPU cost, the reward-repair story is not worth its compute, and we will say so.

Why not just add a stronger KL penalty or stop RL early? Both tactics delay hacking by limiting how far the policy moves. They also cap how far it can improve. We will sweep the KL coefficient and the stopping point for the static-reward baseline and report its best human-judged quality under that tuning budget.

Proposed Experiments

The main testbed is DailyDialog under the GRUE setup in RL4LMs, with a T5-base-sized policy so that several rounds fit in a modest GPU budget. Reward models are initialized from UniEval and from a DialogRPT-style scorer. For transfer, we use two sets of human ratings as held-out test data. The first is the knowledge-grounded USR-TopicalChat ratings. The second is RADE's human-rated EmpaDial and PersonaChat sets from RADE, where RADE reports a Pearson correlation of 0.863 on EmpaDial using references; our reference-free models are compared against that as a ceiling rather than a target.

Our metrics fall into three groups. For human quality, we collect blind pairwise and 1–5 ratings of policy outputs by annotators who never did any editing. For evaluation, we use Mixture-of-Judges scores as an independent automatic judge. For the reward model itself, we measure attack success on the full four-class suite and correlation with human ratings on the held-out sets. BLEU-4 is reported only for continuity with Teacher Forcing Recovers Reward Functions for Text Generation, which reported 4.30.

The decisive check is a curve of human-judged quality and reward-model score against RL steps for each condition. For static rewards, we predict the two curves will separate: the reward score keeps rising while human quality flattens or falls. For the edit loop, we predict they will stay together. A second decisive check is the held-out attack class. A third is the automation-bias audit: if annotators pass more than a small fraction of planted bad replies, the pairs are contaminated and the loop needs a second-reviewer step, which we would cost and report.