Minimally Invasive Preference Updates for Language Models Under Drift

When the people a language model serves change what they want, say by preferring terser answers this quarter or weighting caution over helpfulness in one domain, the model has to be updated. Current methods handle this by weighting recent feedback more heavily and retraining. Nobody checks what that update does to the many prompts whose preferences did not change. Work on two-sided matching markets shows the danger clearly. When one participant leaves, recomputing an exactly stable matching reassigns about 38% of the remaining agents. Tolerating a small fraction of "blocking pairs" (pairs of agents who would rather be matched to each other than to their assigned partners) keeps the new matching close to the old one (Theory of and Experiments on Minimally Invasive Stability Preservation in Changing Two-Sided Matching Markets). Deep reinforcement learning shows a similar effect inside neural networks: a single gradient step flips the preferred action in roughly 10% of states (The Phenomenon of Policy Churn).

We propose to carry the matching-market idea of approximate stability, a guarantee that at most a small, chosen fraction of outcomes are allowed to break, over to preference alignment. Each time the model is updated on drifted preferences, a small budget caps how many prompts may change the meaning of their answers without evidence that preferences changed there. "Meaning" is measured with sentence embeddings, so a harmless rewording does not count against the budget. In two sentences: learn the new preferences where they moved, and on every other prompt allow semantic change in at most an ε fraction of cases. Whether semantic measurement and drift-localization beat the obvious fix, a KL penalty toward the previous model, is the central empirical question.

Core Claim

Time-discounted alignment methods such as Non-Stationary Direct Preference Optimization, or NS-DPO (Right Now, Wrong Then), are the standard tool here. NS-DPO reweights each preference pair by $\gamma^{T-t_i-1}$ according to its age and has good regret guarantees. It applies that discount uniformly, however, and it says nothing about collateral change on prompts whose preferences are stationary.

We claim three things. First, this collateral change is real and measurable in LLM alignment, which no one has shown; the churn paper studied only Atari-style discrete actions. Second, it can be capped by an approximate-stability constraint that counts only unjustified semantic change. Third, the cap costs little or nothing in adapting to the preferences that did drift.

The contribution is distinct from each neighbour:

  • Minimally invasive matching works over discrete rankings with full visibility of the new preferences. We work over continuous output distributions learned from feedback.
  • Con-RAG (Improving Consistency in Retrieval-Augmented Systems with Group Similarity Rewards) enforces invariance across paraphrased queries at a single point in time. We enforce invariance across time and deliberately release it where drift is detected.
  • APLOT (APLOT) uses semantic distance to shape reward-model margins on static data. We use semantic distance to measure and bound policy displacement between updates.

Research Program

The update rule. Let $\pi_{t-1}$ be the deployed model and $\pi_t$ the model being trained at round $t$. Let $\mathcal{A}$ be a set of anchor prompts drawn from past traffic, and let $e(\cdot)$ be a frozen sentence encoder.

For a prompt $x$, define its semantic displacement $D_t(x)$ as the expected embedding cosine distance between answers sampled from $\pi_t$ and from $\pi_{t-1}$.

Define a drift score $s_t(x)$: the change, between the older and the recent feedback window, in the preference margin predicted for $x$'s embedding neighbourhood by a lightweight reward head. Following Beyond Markovian Drifts, clicks/accepts and skips/rejections are tracked separately so that active suppression of a topic registers as drift.

A prompt is blocking when $D_t(x) > \tau$ while $s_t(x) < \delta$. That is, its meaning moved although no evidence says its preferences did. Here $\tau$ and $\delta$ are fixed thresholds, calibrated on held-out paraphrase pairs and on known-stationary prompts respectively.

We train

$$\min_{\pi_t}\ \mathcal{L}_{\text{NS-DPO}}(\pi_t) \quad \text{subject to} \quad \Pr_{x\sim\mathcal{A}}\big[D_t(x)>\tau,\ s_t(x)<\delta\big] \le \epsilon.$$

The constraint is relaxed to a hinge penalty $\lambda\,\mathbb{E}_{x\in\mathcal{A}}[\mathbb{1}(s_t(x)<\delta)\max(0, D_t(x)-\tau)]$. The multiplier $\lambda$ is raised or lowered by dual ascent until the measured blocking fraction sits at $\epsilon$.

Why the budget is nonzero. Setting $\epsilon>0$ is deliberate. The churn paper reports that freezing a policy hurts learning because churn supplies exploration, so we tolerate some change rather than forbid it. This mirrors the 0.5%–5% blocking tolerance in the matching experiments.

Keeping it cheap. Semantic displacement is estimated from a few samples per anchor prompt with a cached encoder. This avoids the optimal-transport and ensemble costs that make semantic rewards expensive.

Measurement study (first deliverable). Before any method, we measure semantic preference churn under NS-DPO and DPO update sequences. This is the fraction of stationary prompts whose answer meaning changes, and whose preferred answer under the round-$(t{-}1)$ judge flips, after each round. If this churn turns out negligible, the proposal stops here, and that null result is itself worth reporting.

Testbed with known ground truth. Existing drift sets drift everything at once, so we build one where only part of the data drifts. We take UltraFeedback, which scores responses separately on instruction-following, honesty, truthfulness and helpfulness. We then shift the weighting of those aspects over time only for a subset of prompt domains, leaving the other domains stationary. This gives known drifted and known stationary prompts. We also use TV-HH, where helpfulness–harmlessness priorities shift suddenly or gradually, and 2C NSGO, where regional opinions drift. For stationarity on these two sets we hold out capability prompts (reasoning, factual QA) and safety prompts unrelated to the drift. Models follow NS-DPO: Llama-3.2-1b, with Llama-2-7b for the main comparisons.

Baselines

Naive references. The naive reference is standard DPO trained on all accumulated data. On UltraFeedback-LM, NS-DPO reports this reaching an AlpacaEval 2.0 length-controlled win rate (LCWR) of 7.29. Under preference flips it falls to roughly 50–55% reward accuracy. Alongside it we run sliding-window DPO, which discards all but the most recent window.

Strongest existing method: NS-DPO (Right Now, Wrong Then). It is the only method in the file designed for exactly this drift setting. Its reported results are an LCWR of 8.93 on UltraFeedback-LM, more than 10% higher reward accuracy than baselines under gradual drift on TV-HH, and reward accuracy held above 60–70% under flips on UltraFeedback-RM. We reuse its $\gamma$ grid, and our method keeps NS-DPO's loss so that only the constraint differs.

The real competitor: NS-DPO plus a token-level KL penalty toward $\pi_{t-1}$ on the same anchor set, with the coefficient tuned over the same grid. This is continual-learning distillation and is the standard fix. A second variant adds experience replay of old preference pairs.

Two ablations isolate the key ingredients. Drift-gated token KL keeps our gating but replaces semantic displacement with KL, which tests whether semantic measurement matters. The ungated semantic penalty applies the displacement penalty to all anchor prompts, which tests whether drift localization matters.

Fair comparison. All methods get identical data streams, the same number of gradient steps per round, and the same anchor set. Each gets an equal hyperparameter budget of twelve configurations per method, selected on a validation stream.

What counts as winning. At matched current-preference reward accuracy (within one point of tuned NS-DPO), our method must show clearly lower semantic churn on stationary prompts, with flipped-preference rate as the secondary measure. It must also hold AlpacaEval LCWR at or above NS-DPO's. The win must hold across the whole accuracy–churn trade-off curve traced by sweeping each method's regularization strength, not at a single point.

Why not just…?

Why not just add a KL penalty toward the previous model? This is the serious threat. Token-level KL cannot tell a harmless rewording from a change of meaning, so it spends its budget resisting paraphrase. When applied everywhere, it also resists change on the drifted prompts that must change. We expect it to trade drift-adaptation accuracy for stability on a worse curve. The drift-gated-KL ablation separates the two reasons. If gated KL matches our method, the contribution reduces to drift localization, and we would report it that way rather than claim semantic measurement matters.

Why not just tune NS-DPO's discount or retrain on a sliding window? Both act on which data is used, not on where the model moves. Neither can protect a stationary math prompt from side effects of a tone update. Stationary knowledge is also shared structure worth keeping, as argued in A Shared Low-Rank Adaptation Approach to Personalized RLHF. The measurement study directly tests whether better discounting alone reduces churn.

Why not just replay old preference pairs? Replay preserves old preferences only where old pairs exist and still hold. Under drift, replaying stale pairs on drifted domains fights the update, which is exactly the failure NS-DPO was built to avoid. Replay is run as a baseline. If replay with discounting achieves our churn levels, the constraint is unnecessary.

Why not just use in-context history with a frontier model? Frontier models reach under 30% perfect outcomes on AcCoRD (AcCoRD). In-context adaptation also offers no control over stationary behaviour, so it is out of scope as a training-time competitor.

Decisive Checks

The first check is the measurement study. Standard NS-DPO updates must produce non-trivial semantic churn on known-stationary UltraFeedback domains and on held-out TV-HH capability prompts.

The second is the trade-off curve between current-preference accuracy and stationary churn, compared against NS-DPO+KL and both ablations.

The third is an ε sweep. Setting ε to zero should measurably slow adaptation on drifted prompts, confirming the churn paper's finding that forbidding change carries a cost. A moderate ε should recover NS-DPO's adaptation.

The fourth is a guard against gaming the embedding space. A policy could keep answers close in embedding space while changing their facts, echoing the paraphrase fragility documented in Preference Instability in Reward Models. To catch this, stationary-prompt answers are also scored for factual agreement by a judge model and exact-match QA, independent of the encoder.