Telling Changed Words from Changed Minds in Drifting Preference Data

Systems that learn from human preferences over time, such as language models tuned on pairwise "this answer is better than that one" judgments, now commonly assume that old judgments go stale. The standard fix is to down-weight old comparisons exponentially. That fix silently assumes something false in most real deployments: that the data's vocabulary stays put. In practice the way items are described also changes over time and across sources. Product catalogs rename attributes and switch units. Hospitals change lab codes and recording cadences. Annotation guidelines get reworded. When an old comparison disagrees with today's model, it may be because people now want something different. It may also be because the old record describes the same thing in different terms. Discounting by age cannot tell these apart, so it throws away valid evidence whenever the representation shifts, even if nobody's preferences moved.

We propose to separate the two explicitly. A small per-era translation layer, trained only on unlabeled inputs, maps old records into today's representation. A preference model with time discounting then handles only the remaining drift, which is real preference change. The identifying idea is simple. A change in representation shows up in the inputs alone, with no labels needed. A change in preference shows up only in the labels, given fixed content. If this split works, old data that was merely described differently becomes usable again. Data whose labels genuinely flipped is still discounted.

Core Claim

When preference data is collected across eras or sources that describe items differently, discounting old comparisons by time alone conflates two kinds of change. A method that first aligns each era's representation using unlabeled data, and only then applies time discounting, should do three things. It should match plain discounting when only preferences drift. It should clearly beat plain discounting when only representations drift. It should beat both plain discounting and representation normalization alone when both happen at once. This regime appears in the file's problem descriptions but is not evaluated anywhere in it, so this proposal must build the benchmark as well as the method.

Where This Sits

The temporal side supplies the method we extend and the failure we target. Right Now, Wrong Then introduces Non-Stationary Direct Preference Optimization (NS-DPO). It is standard direct preference optimization, a way to tune a model directly on pairwise comparisons without a separate reward model, with each comparison weighted by $\gamma^{T-t-1}$ for its age. The paper shows that stationary training collapses toward chance when preferences flip. Its weighting is uniform across all comparisons from a given time, and its best $\gamma$ depends on a drift budget that is rarely known. We keep its loss but change what "old" means: age is measured after aligning representations, not in raw input space.

A Shared Low-Rank Adaptation Approach to Personalized RLHF supplies the structural template. It splits a low-rank adapter into a shared part and per-user parts. We split along a different axis, into per-era input translation and a shared, slowly drifting preference head. Their per-user blocks are static and they assume one shared vocabulary.

Adapting Job Recommendations to User Preference Drift with Behavioral-Semantic Fusion Learning is the closest crossing. It uses job-description text to keep drifting click sequences from over-reacting. Its semantic space is a single fixed schema, and it never asks whether the semantics themselves moved. Towards Explainable Conversational Recommendation likewise balances long-term and in-session preferences over a fixed, clean concept graph.

The heterogeneity side motivates the problem and supplies the alignment tools. Record2Vec turns site-specific ICU time series into language and then into vectors, and transfers across hospitals well. INPUTADAPTER learns a translation in front of a frozen predictor. Its finding constrains our design: the best translation moved target data farther from the source in Wasserstein distance. So our alignment objective cannot simply minimize distributional distance. It must preserve correspondence of content, using paired or anchor items, rather than matching marginals. Both papers address only fixed prediction targets. The federated EHR scoping review documents "hidden heterogeneity," where identical labels mean different clinical criteria across sites. This is exactly the case that text normalization leaves untouched, and our benchmark includes it on purpose.

Method

Each comparison $(x, y_w, y_l, t)$ carries an era (or source) index $e(t)$. We learn a light translation $\phi_e$ per era. In the text setting it is a small adapter on the encoder. In the structured setting it is a learned remapping of fields, units, and codes. Each $\phi_e$ maps era-$e$ inputs into the current era's representation. It is trained without preference labels, using three ingredients: anchor items that appear in both eras under different descriptions, a cycle-consistency term, and co-occurrence statistics in the spirit of the multi-center embedding alignment in KESER. The preference model is then trained with

$$ \mathcal{L}(\theta) = -\sum_{(x,y_w,y_l,t)} \gamma^{T-t-1}\,\log\sigma\!\Big(\beta\,\big[\Delta_\theta(\phi_{e(t)}(x,y_w)) - \Delta_\theta(\phi_{e(t)}(x,y_l))\big]\Big), $$

where $\Delta_\theta(\cdot)$ is the policy-to-reference log-likelihood ratio, as in NS-DPO, $\beta$ is the usual temperature, and $\gamma$ is chosen on held-out recent data.

A second variant replaces the single $\gamma$ with a per-era weight estimated from disagreement. After translation, we measure how often era-$e$ comparisons disagree with a model fit on the most recent era. Eras whose disagreement is explained by translation keep high weight. This directly targets NS-DPO's unknown-budget problem.

The method rests on one identifiability assumption: representation change is visible without labels and preference change is not. We test this assumption rather than assume it. Two benchmark conditions are built to violate it, and we report where the method breaks.

Research Program

The main artifact is a factorial benchmark. It crosses preference drift (none, gradual, abrupt) with representation drift (none, surface renaming, unit and scale changes, and "meaning shift," where the same words acquire different referents).

The text arm builds on the datasets from NS-DPO: TV-HH, UltraFeedback-RM/LM, and 2C NSGO. We inject representation drift into later eras by systematically rewriting prompts and responses. Examples are renamed attribute terms, changed number formats and units, and, for meaning shift, relabeled scale words such as moving the price band that "budget" refers to. Ground-truth preferences are held fixed or flipped independently of these rewrites. The structured arm uses product attributes from the WebShop catalog underlying AcCoRD-Shop, with era-specific attribute schemas and simulated users whose weights drift.

A smaller clinical arm uses MIMIC-IV and eICU, the site pair used by INPUTADAPTER, processed with the MIMIC-IV pipeline. There, synthetic clinician preferences over pairs of alert timings drift over time, while the two sites differ in codes, units, and cadence. This arm tests whether the separation survives realistic site heterogeneity, not just synthetic rewriting. Its preference labels are simulated, and we will say so plainly.

Baselines

The naive reference point is standard DPO on all pooled data with no discounting and no alignment. NS-DPO's own results already establish that it fails under drift. In the representation-only conditions, however, it is a meaningful upper bound on what the old data contains.

The strongest existing method is NS-DPO from Right Now, Wrong Then. On the original, representation-stable versions of its benchmarks, it reports reward accuracy remaining above 60–70% under abrupt drift, where DPO falls to roughly 50–55%. It reports more than 10% higher reward accuracy than stationary baselines under gradual drift, and a length-controlled AlpacaEval 2.0 win rate of 8.93 versus 7.29 for DPO on UltraFeedback-LM. We first reproduce these numbers on the unmodified datasets as a sanity check, then run NS-DPO on every factorial cell. It receives the same base model, the same data, and the same $\gamma$ search grid and selection protocol as our method. The file contains no reported result for any method on preference data that has both temporal drift and representation drift, so every number in the crossed cells will be new.

The second strong competitor is normalize-then-discount. Every era is first converted into a common substrate, either by LLM rewriting into a canonical description in the manner of Record2Vec or by an explicit schema mapping, and then NS-DPO is applied. This competitor receives the same LLM access and compute we spend on training the translation layer.

The key ablation removes the label-free constraint. The translation layers are trained jointly with the preference loss, so either component can absorb any change. If this ablation matches the full method, the identifiability story is wrong.

Beating the baselines means four things. Reward accuracy and win rate must be statistically indistinguishable from NS-DPO in preference-only cells. They must be higher in representation-only cells, approaching pooled DPO. They must exceed both NS-DPO and normalize-then-discount in the meaning-shift cells. All of this is judged over at least three seeds.

Why not just…?

Why not just tune $\gamma$ in NS-DPO more carefully? In a representation-only cell, the optimal $\gamma$ is 1, because nothing about preferences changed. In a mixed cell, a single $\gamma$ must trade forgetting stale labels against keeping relabeled-but-valid data. No single scalar can do both. The factorial design shows this directly by sweeping $\gamma$ to its oracle value in every cell. If oracle-tuned NS-DPO matches us in mixed cells, our contribution reduces to a $\gamma$-selection heuristic, and we will report that.

Why not just normalize everything into text first, as Record2Vec does, and then discount? This is the real threat. On surface renaming and unit changes it may well tie with our method, and we expect to report that honestly. It should fail on meaning shift, where the words are unchanged but their referents moved. This is the scoping review's hidden heterogeneity, and a faithful rewriter preserves exactly the ambiguity that causes the error. If normalize-then-discount also wins the meaning-shift cells, the separate translation layer is unnecessary and the proposal shrinks to a benchmark result.

Why not just retrain on a recent sliding window? Windowing discards all old data regardless of why it disagrees, and it is especially costly when recent eras are small. Record2Vec's few-shot results and KESER's reliance on pooling across very large cohorts both indicate that small recent slices are often insufficient. We include window-only training at matched recent-data sizes. We expect it to lose most in representation-only cells, where all the old data was still valid.

Why not just learn per-example weights with a robust or noise-aware loss? Such weights can down-weight disagreeing comparisons. They cannot recover them, because a relabeled comparison stays misaligned and is merely ignored. The ablation of our method with translation disabled but learned per-era weights kept isolates this contribution.

Decisive Checks

The first check is a crossover pattern on the text arm. The full method should tie NS-DPO when only preferences drift and clearly beat it when only representations drift. Without this pattern the core claim fails.

The second check is the meaning-shift cells. Here the method must beat normalize-then-discount, or the separate translation layer has no justification.

The third check is a diagnostic. After training, the per-era disagreement estimates should correlate with the injected preference-flip schedule and not with the injected rewrite schedule. This is the most direct evidence that the method separated changed words from changed minds.

Finally, on the MIMIC-IV/eICU arm, we report whether these patterns persist under real site differences. A failure there would show that the label-free alignment assumption breaks on real clinical heterogeneity. That would be a useful negative result, not a footnote.