Satisficing Under Preference Drift: Choosing How Much History to Trust Without Guessing the Drift Rate

When people's preferences change over time, a model trained on old feedback learns things that are no longer true. The standard fix is to down-weight old feedback with a decay rate, but the right rate depends on how fast preferences are changing, which nobody knows in advance. We propose a different rule. First, require the model to reach an acceptable target on the freshest feedback. Then, among models that meet that target, pick the one whose loss rises most slowly as older feedback is added back in. Preferences that have held steady get learned from the whole history, because agreeing with old data costs nothing. Preferences that have flipped get learned from recent data only, because old data cannot be fit without missing the target. No decay rate has to be chosen.

The idea carries over robust satisficing to preference optimization. Robust satisficing is a decision rule that asks for performance above a user-set target $\tau$ and, subject to that, minimizes fragility, meaning how fast performance degrades per unit of distribution shift. The novelty is the choice of shift: we measure it along time, as nats of distance from "only the most recent feedback" to "all feedback". This turns the unknown drift rate from a hyperparameter into something the training objective handles directly.

Core Claim

Offline preference learning under drift, as in Direct Preference Optimization (DPO, which fits a policy directly to pairwise "A is better than B" labels), currently commits to one view of how much history to trust. Right Now, Wrong Then: Non-Stationary Direct Preference Optimization under Preference Drift introduces NS-DPO, which weights each sample by $\gamma^{T-t_i-1}$. This motivates our work, because it shows drift-aware weighting matters: plain DPO falls to roughly 50–55% reward accuracy under preference flips, while NS-DPO stays above 60–70%. It also constrains our work, because the paper says the best $\gamma$ depends on a variation budget that is unknown at deployment.

Our claim is that a fragility objective over a family of time windows matches a well-tuned NS-DPO when the drift rate is the one it was tuned for, and degrades much less when the true drift rate is faster, slower, or different across preference attributes. It should also keep higher accuracy on attributes that never drifted, because it does not throw away consistent old data.

The Objective

Order the $n$ preference pairs by timestamp. Let $P_k$ be the uniform distribution over the most recent $k$ pairs, and let $L_k(\theta)$ be the average DPO loss of policy $\theta$ under $P_k$. Fix a smallest window $k_{\min}$, the "freshest data" on which we demand the target. Since the support of $P_{k_{\min}}$ lies inside the support of $P_k$, the divergence $\mathrm{KL}(P_{k_{\min}}\,\|\,P_k)=\log(k/k_{\min})$ is finite and measures, in nats, how far a window reaches into the past. We define fragility as the steepest loss increase per nat of added history:

$$\kappa_\tau(\theta)=\max_{k_{\min}<k\le n}\frac{L_k(\theta)-\tau}{\log(k/k_{\min})},\qquad \text{minimize } \kappa_\tau(\theta)\ \text{ subject to } L_{k_{\min}}(\theta)\le\tau .$$

This is the same structure as Iterative Robust Satisficing: Minimizing Performance Degradation Under Distribution Shift. IRS keeps training loss below $\tau$ while minimizing the slope of degradation over exponential tilts of the training distribution. It reduces the search to one scalar and uses Danskin's theorem, which says the gradient of a maximum is the gradient at the maximizing argument. That gives an update at the cost of ordinary stochastic gradient descent.

We reuse that machinery. Our family is also one-dimensional, indexed by $k$. Given the maximizing window $k^\star$, the gradient is a DPO gradient over the last $k^\star$ pairs, scaled by $1/\log(k^\star/k_{\min})$. The constraint is enforced with a Lagrange multiplier. We track $L_k$ for all $k$ by caching each example's most recent loss and keeping prefix sums, so the search over $k$ costs almost nothing per step. A smooth version replaces windows with exponentially discounted distributions. We will test both, because exponential discounting is NS-DPO's own family, and that makes the comparison very direct.

The IRS paper notes that its guarantees cover reweightings within the training support only. Our time-window family stays inside that support. Extrapolating to future drift is outside it, and we evaluate that case separately below rather than claim it.

Two design questions remain open. The first is $\tau$. Robust Satisficing Gaussian Process Bandits Under Adversarial Attacks and Robust Bayesian Satisficing warn that guarantees collapse when the target cannot be reached. Preference loss has a natural reference point, though: $\log 2$ corresponds to chance-level ranking. We therefore express $\tau$ as a target reward accuracy on the freshest window. We start it loose and tighten it with IRS-style threshold scheduling, backing off whenever the constraint stays infeasible. The second question is $k_{\min}$. It plays a role similar to $\gamma$, so the burden is on us to show that results are much less sensitive to $k_{\min}$ than NS-DPO's are to $\gamma$. If they are not, the proposal fails.

Research Program

We first characterize the objective analytically in a two-attribute Bradley–Terry model, the standard model in which the probability of preferring A over B is a logistic function of their reward difference. One attribute is stable and one flips at an unknown change point. The question is whether the minimizer keeps the full-history estimate for the stable attribute and the recent-window estimate for the flipped one, and how $\tau$ trades peak recent accuracy against degradation when a change turns out to be transient.

We then implement the method on top of DPO at the scale used in the NS-DPO experiments. Finally, we test the variant that also tilts by per-example loss, as IRS does. This is the most direct answer to NS-DPO's limitation that every sample at the same timestamp gets the same weight.

DRO-REBEL: Distributionally Robust Relative-Reward Regression for Fast and Efficient LLM Alignment robustifies alignment over Wasserstein, KL, and $\chi^2$ balls of a fixed radius. It shows that coverage requires large radii, which slow convergence to $O(n^{-1/4})$. This motivates replacing the radius with a slope. We will also check whether that rate penalty shows up empirically in the time family.

Lightweight Robust Direct Preference Optimization puts a $\chi^2$ ball around preference labels to handle stationary label noise. Its target is different: it treats flips as random corruption, whereas we treat them as a systematic change in time. We include it to show that noise-robustness alone does not solve drift.

Baselines

The naive reference point is plain DPO on the full, unweighted history. Any drift-aware method must clearly beat it on post-drift test pairs. A sliding-window DPO trained only on the last $k_{\min}$ pairs isolates the value of using older data at all.

The strongest existing method is NS-DPO from Right Now, Wrong Then. On UltraFeedback-LM it reports an AlpacaEval 2.0 length-controlled win rate of 8.93 against 7.29 for DPO. Under preference flips it reports reward accuracy above 60–70%, against roughly 50–55% for DPO. We run NS-DPO in two ways. The fair version tunes $\gamma$ once, at a nominal drift condition, with the same number of tuning trials we give to $\tau$ and $k_{\min}$, and then keeps it fixed. The oracle version retunes $\gamma$ for every drift condition, which gives an upper envelope.

Beating NS-DPO means two things. Averaged over the drift sweep, we must have higher worst-case and mean post-drift reward accuracy than fixed-$\gamma$ NS-DPO, with matched data, base model, steps, and tuning trials. We must also stay within a small margin of oracle NS-DPO at its own tuned condition.

A fixed-radius KL-DRO over the same time family stands in for budget-driven robustness, following DRO-REBEL's KL formulation. We found no reported result on Lacuna for DRO-REBEL on TV-HH, UltraFeedback drift splits, or 2C NSGO. DPO-PRO is run with its published noise settings.

Two ablations isolate the key ingredient. The first is IRS applied to the DPO loss with loss-tilts only and no time family. The second is our objective with the fragility term removed, which leaves only the satisficing constraint on the freshest window. If the first ablation matches the full method, time is not doing the work. If the second matches, fragility is not doing the work.

Why not just…?

Why not just tune $\gamma$ on a held-out slice of recent data? This is the most serious threat. If the most recent slice predicts the deployment distribution well, validation-tuned NS-DPO could match our method at lower complexity. We include it as a baseline with the same validation data. It should be weakest when drift keeps changing after the validation slice, or when attributes drift at different rates so that no single $\gamma$ fits them all. If it ties us under both conditions, we will report that the fragility objective adds nothing beyond validation.

Why not just average NS-DPO over a grid of $\gamma$ values? Averaging hedges over the drift rate by spreading weight uniformly across the decay family, whereas fragility guards against the worst slope. These could behave the same in practice. We run a $\gamma$-mixture baseline over the same grid our smooth variant searches. The deciding measurement is accuracy on the stable attribute versus the flipped attribute separately: our objective predicts the mixture will dilute the flipped attribute while we will not.

Why not detect the change point and retrain on post-change data? For a single sudden flip, this is probably competitive and may win outright. We include a change-point-then-retrain baseline. We expect it to lose under gradual drift, where there is no clean break, and under attribute-specific drift, where retraining on post-change data throws away stable old preferences. If it wins on gradual drift too, our core claim is wrong.

Why not use worst-case DRO with a large radius? DRO-REBEL's own analysis gives large radii a slower rate. The satisficing literature shows that robust optimization accumulates linear regret when the budget is misestimated (Robust Bayesian Satisficing). The fixed-radius KL baseline tests whether those findings carry over to temporal preference data.

Proposed Experiments

We use the drift benchmarks from NS-DPO. TV-HH shifts the balance between helpfulness and harmlessness over time. UltraFeedback-RM and UltraFeedback-LM flip preferences at change points. 2C NSGO moves between US-centric and Japan-centric viewpoints. We add controlled variants of each.

The decisive experiment is a drift-rate sweep. Every method's hyperparameters are fixed at one nominal drift rate. We then vary the true rate across sudden, gradual, and accelerating schedules, and plot post-drift reward accuracy against the true rate. The prediction is that fixed-$\gamma$ NS-DPO peaks at its tuned rate and falls off on either side, while our curve is flatter, giving up a little at the peak.

The second check uses heterogeneous drift. In TV-HH, only the harmlessness preference flips while helpfulness stays fixed, and we report accuracy on each attribute separately. The third check measures sensitivity: the spread in accuracy across a grid of $k_{\min}$ values, compared with the spread across an equivalent grid of $\gamma$ values.

We also test future-drift extrapolation. Training stops at time $T$ and we evaluate on pairs from later times while drift continues. This is honestly outside the method's support-based guarantee, so we report it as an open stress test rather than a claim. Generation quality is measured by AlpacaEval 2.0 length-controlled win rate on UltraFeedback-LM, against NS-DPO's 8.93. Wall-clock time is compared with DPO, because IRS reports a 1.26x speedup over its plain baseline (ERM) from skipping samples that already meet the target.

The proposal succeeds if the drift sweep and the per-attribute split both favor it over fixed-$\gamma$ NS-DPO, validation-tuned $\gamma$, and change-point retraining. It fails if either of the latter two matches it everywhere.