Where Did the Preferences Move? Locally Calibrated Forgetting for Drifting Reward Models

When people's preferences change over time, a reward model trained on last month's judgments keeps giving confident answers, and many of them are now wrong. Current tools miss this failure for a structural reason. Methods that detect out-of-distribution inputs ask whether a prompt or response looks unfamiliar. Under preference drift, though, the prompts often look exactly like the training data. What changed is which answer people prefer. A reward model can be confidently wrong on perfectly familiar inputs, and no input-novelty detector will notice. This note proposes measuring drift where it actually shows up: in the gap between the reward model's stated confidence and how often its judgments agree with fresh human labels, computed separately for each region of prompt space. That local gap then decides how fast old preference data is forgotten in each region.

In two sentences: preference drift is usually partial, so safety judgments may flip while coding-style preferences stay put, and a single global forgetting rate either throws away still-valid data or holds on to stale data. We estimate overconfidence locally from a trickle of recent labels, use it to give each region its own forgetting rate during preference optimization, and use the same signal to widen the reward model's uncertainty exactly where it has become unreliable.

Core Claim

The claim has two parts, and each can be falsified. First, under drift in what people prefer (as opposed to drift in what inputs look like), input-space uncertainty signals barely separate drifted pairs from stable ones. This covers feature-prediction error, nearest-neighbor distance and ensemble disagreement. A local calibration residual, meaning confidence minus agreement with recent labels among nearby examples, separates them well. Second, when drift affects only part of the prompt space, per-region forgetting driven by this residual beats time-discounting with the best single global rate. It keeps accuracy in stable regions and recovers faster in drifted ones.

Relation to Prior Work

The direct ancestor is Right Now, Wrong Then: Non-Stationary Direct Preference Optimization under Preference Drift. NS-DPO weights each historical preference pair by $\gamma^{T-t_i-1}$ in a direct preference optimization loss, and proves regret bounds for drifting environments. It supplies our training objective. It also exposes the gap we target. Its discount is global and blind to timestamps' content, so a recent noisy pair is trusted simply for being recent. The best $\gamma$ also depends on how much the preferences vary in total, which is unknown. We keep its form but make the rate depend on local evidence of drift.

A second group handles pessimism about uncertainty but assumes stationarity. Uncertainty-Penalized Direct Preference Optimization penalizes pairs by the variance of a five-member reward-model ensemble on static Anthropic-HH data. From Curiosity to Caution subtracts a feature-prediction error from the reward to flag atypical Best-of-N candidates. Both measure input unfamiliarity, and that is exactly the signal we expect to fail under drift in what people prefer. Two further results constrain this family. From Risk to Uncertainty shows that ensemble members sharing an architecture extrapolate in the same direction, so their disagreement collapses. Where are we with calibration under dataset shift shows that temperature scaling fitted before a shift does not survive it. We treat these as reasons to measure calibration against fresh outcomes rather than infer it from model internals. The same logic rules out static-manifold regularizers such as Calibrated Language Model Fine-Tuning for In- and Out-of-Distribution Data, whose off-manifold points become obsolete once the preference boundary moves.

The closest work on monitoring drift is CARE: Adaptive Calibration for Reliable Recommendations. CARE tracks loss discrepancy with Bayesian change-point detection and resizes conformal recommendation sets. It supplies the idea of loss-based monitoring. It differs from our proposal in three ways. It detects drift globally in time, not locally in input space. It adapts prediction sets rather than the training weights of a generative alignment objective. It inherits the overconfidence of the underlying model rather than correcting it.

Generalizing Reward Modeling for Out-of-Distribution Preference Learning and CausPref study shift across static domain splits, such as product categories, subreddits and cities, and they do not output calibrated confidence. Our proposal studies ongoing drift and makes confidence its central quantity. For the nearest-neighbor machinery we borrow from Far Away in the Deep Space, which reports that nearest-neighbor search in transformer features holds up well. We use neighbors only to define locality, not as a novelty score.

Mechanism

Let $z_i$ be the reward model's final-layer embedding of prompt $x_i$. Let $p_i=\sigma(r(x_i,y_w)-r(x_i,y_l))$ be the reward model's predicted probability that the labeled winner is preferred. From the most recent labeled time slice we take the $k$ nearest neighbors $N_k(z)$ of any point. We define the local overconfidence as

$$g(z)=\frac{1}{k}\sum_{j\in N_k(z)}\Big(\max(p_j,1-p_j)-\mathbb{1}[p_j>0.5]\Big),$$

which is the average stated confidence minus the average agreement with recent human labels near $z$. This quantity is near zero where the reward model is still calibrated and grows where preferences have moved. Each old pair then gets a local discount rate $\gamma_i=\exp(-\alpha\, \max(g(z_i),0))$. It enters the NS-DPO weight as $\gamma_i^{T-t_i-1}$, where $\alpha$ is a single scale parameter. The same $g$ is used at inference. Its role is to shrink reward margins toward indifference in proportion to local overconfidence, so the model reports high uncertainty exactly where it has become stale.

The design addresses the known failure modes directly. It replaces NS-DPO's unknown variation budget with a quantity estimated from data. Because $g$ averages over neighbors, a single noisy recent pair cannot trigger forgetting. Because $g$ is conditioned on outcomes, it does not depend on ensemble disagreement.

Research Program

We first run a diagnostic study. On drifting streams, we compare how well four signals rank post-drift pairs by whether the reward model now gets them wrong. The four signals are Caution-style feature-prediction error, ensemble variance, nearest-neighbor distance to training data, and $g$. This tests the first half of the claim independently of any training.

Next we build partial-drift benchmarks. TV-HH already separates helpful and harmless preferences. We construct variants in which only the harmlessness subset flips, or only a fraction of topics flips. We do the same for 2C NSGO, where only some question categories move from US-centric to Japan-centric answers.

We then train reward models and policies with locally discounted NS-DPO. We compare global-drift, partial-drift and no-drift conditions, the last to check that the method does not forget when nothing moved.

Finally, we measure the label cost. This means how many recent labeled pairs per time slice are needed before $g$ becomes reliable, which addresses the sparsity concern raised in the drift literature.

Baselines

The naive reference point is standard DPO on the pooled stream. On TV-HH with flipped preferences, NS-DPO's authors report that its reward accuracy falls to "near-random levels (~50-55%)" (Right Now, Wrong Then).

The strongest existing method is NS-DPO. The same paper reports it "maintained robust reward accuracy above 60-70%" in the flipped scenario, more than 10% above baselines under gradual drift, and a length-controlled AlpacaEval 2.0 win rate of 8.93 versus 7.29 for DPO on UltraFeedback-LM. We will run it with the same base model, data order and training steps as our method. Its $\gamma$ will be tuned over the same grid and the same number of trials as our $\alpha$ and $k$, on a held-out slice of the future stream. We also report an oracle-$\gamma$ version tuned on test data as an upper bound for any global discount. Beating NS-DPO means higher post-drift reward accuracy and lower post-drift calibration error on partial-drift streams than oracle-$\gamma$ NS-DPO, while staying within noise of it on global drift.

Uncertainty-penalized DPO, run with a same-size ensemble, is the strongest pessimism competitor. No reported numeric result was found on Lacuna for its mean reward on Anthropic-HH. For Caution, no reported result was found on Lacuna for any preference-drift benchmark. For CARE, no reported number was found in the file for comparable settings. We will run all three ourselves.

Two ablations isolate the key ingredient. The first replaces $g$ with an input-novelty score, keeping locality but dropping outcome-conditioning. The second replaces $g$ with a single global calibration residual detected CARE-style, keeping outcome-conditioning but dropping locality. If the full method does not beat both ablations, the core claim fails.

Why not just…?

Why not just tune NS-DPO's global discount on a validation slice? This is the most serious threat, and the oracle-$\gamma$ baseline tests it directly. Under global drift we expect a well-tuned $\gamma$ to match our method, and we will report that result honestly. The argument for locality is specific to partial drift. There, any single rate must trade accuracy in stable regions against adaptation in drifted ones. If oracle-$\gamma$ matches us on partial-drift TV-HH too, the proposal reduces to a hyperparameter heuristic and we would say so.

Why not just add ensemble or feature-error pessimism to NS-DPO? Both signals measure how unfamiliar an input looks. When preferences flip on familiar prompts, the inputs stay in distribution. In addition, From Risk to Uncertainty shows that same-architecture ensembles agree while wrong. The diagnostic study and the input-novelty ablation measure this directly. If novelty signals rank drifted pairs as well as $g$ does, our first claim is wrong.

Why not just detect change points globally and retrain on recent data, as CARE-style monitoring would? Retraining after a detected change throws away still-valid data from unaffected regions. Retraining on a recency window is also just NS-DPO with a hard cutoff. The global-residual ablation shows how much of the gain comes from locality rather than from conditioning on outcomes.

Decisive Checks

We evaluate on TV-HH, its partial-flip variants, 2C NSGO with category-restricted drift, and the UltraFeedback-RM change-point stream. The metrics are post-drift reward accuracy, expected calibration error of the reward model computed separately on drifted and stable regions, AUROC for ranking drifted pairs, and AlpacaEval 2.0 length-controlled win rate for policies. Because When Rubrics Backfire shows that LLM judges themselves drift, we treat dataset ground-truth preference labels as primary and judge-based win rates as secondary.

The result that would settle the question is as follows. On partial-drift streams, the full method must exceed oracle-$\gamma$ NS-DPO in stable-region accuracy while matching it in drifted regions. The input-novelty ablation must fail to do the same. On the no-drift control, $g$ must stay near zero so that nothing is forgotten.