Testing for Preference Drift Instead of Discounting by the Clock

Preference-tuned models are trained on comparisons collected over months, and what people prefer changes during that time. The current fix for this drift, Non-Stationary Direct Preference Optimization (Right Now, Wrong Then), down-weights each comparison by a fixed exponential discount in its age. That works when drift is smooth and steady. The paper itself notes the problem: the best discount rate depends on the total amount of drift over the horizon, which nobody knows in advance. A fixed discount also forgets old data at the same rate whether or not anything has changed.

Speech analysis faced a version of this problem decades ago. Speech is mostly slowly varying vowels broken by abrupt events such as plosive bursts. Time-varying autoregressive tests for multiscale speech analysis replaced fixed analysis windows with a statistical test. It merges neighbouring segments when a stationary model explains them as well as a time-varying one, and splits them when it does not. The result is long windows where nothing changes and short windows around transitions. We propose carrying that test over to preference data. A likelihood-ratio test on a cheap time-varying preference model decides which past comparisons still describe current preferences, and preference optimization trains on that adaptively chosen history rather than on an exponentially decayed one. We evaluate on the text drift benchmarks where the discount method was introduced, and on a spoken-preference stream where the criterion judges use to compare voices changes over time.

Core Claim

Tying the down-weighting of old preference data to a change test, rather than to elapsed time, should match exponential discounting under smooth drift and clearly beat it under abrupt or bursty drift. It should also lose nothing when preferences are stationary, all without knowing the amount of drift in advance. The speech literature contributes the mechanism and also a warning. Rudoy et al. found that letting the time-varying model track faster changes, by adding more basis functions, inflates its degrees of freedom and weakens the test. We expect the same trade-off in preference space and make it a measured object rather than a hidden hyperparameter.

Research Program

The building block is a Bradley-Terry model whose reward may change over time. A Bradley-Terry model says a person prefers response $y_w$ over $y_l$ for prompt $x$ with probability $\sigma(r(x,y_w)-r(x,y_l))$, where $\sigma$ is the logistic function and $r$ is a reward. To keep the test cheap, we fit it on frozen embeddings $\phi(x,y)$ from the backbone, using a low-dimensional linear reward $r_t(x,y)=\theta(t)^\top P\phi(x,y)$. Here $P$ is a fixed projection to $d$ dimensions, around 32 to 64. Following the autoregressive-coefficient expansion in the speech test, the weight vector varies in time as

$$\theta(t)=\sum_{k=0}^{q-1} c_k\, L_k(t),$$

where $L_k$ are Legendre polynomials on the segment's time range and $c_k$ are learned $d$-dimensional coefficients. Setting $q=1$ gives the stationary model, which is the null hypothesis. For a candidate merged segment, the test statistic is

$$\Lambda = 2\left[\ell_{q}(\hat c) - \ell_{1}(\hat c_0)\right],$$

the gap in fitted log-likelihood between the two models. It is compared to a chi-squared threshold with $(q-1)d$ degrees of freedom, the standard asymptotic reference for nested likelihood-ratio tests. We also calibrate it by permuting timestamps, because the asymptotics are shaky at small segment sizes.

The procedure works on the time-ordered log in four steps. First, the log is split into small blocks at the native timestamp granularity. Second, blocks are merged bottom-up as in the speech procedure whenever the null is not rejected. Third, the segment containing the present becomes the fully weighted training set. Fourth, each earlier segment receives a weight equal to the held-out agreement between its fitted reward and the current segment's reward, clipped to $[0,1]$. This soft step also addresses a second limitation of discounting. Two comparisons from the same date are no longer forced to share a weight: a block whose preferences still match today's keeps its influence, and a block that contradicts today's loses it. The weights then enter ordinary DPO, or the NS-DPO loss with $\gamma=1$, on the full model.

The study has three parts. The first is the method itself. The second is an analysis of the trade-off between basis order $q$ and test sensitivity, which tells us how fast a drift the test can see before it loses power. The third is a transfer to speech-preference data, where reward drift is less studied but more plausible: listeners adapt to synthetic voices, and the criteria for judging spoken dialogue vary with context.

Baselines

The naive reference is standard DPO on the whole pooled log with uniform weights. Right Now, Wrong Then reports this collapses to roughly 50–55% reward accuracy when preferences flip mid-dataset. Any method that cannot clear this bar is not addressing drift at all.

The strongest existing method is NS-DPO itself. It keeps NS-DPO's training loss but sets the weight of each comparison to $\gamma^{T-t_i-1}$, where $T$ is the current time and $t_i$ the comparison's timestamp. That paper reports reward accuracy staying above 60–70% under drift, an advantage of over 10% in reward accuracy in gradual-drift settings, and an AlpacaEval 2.0 length-controlled win rate of 8.93 against 7.29 for DPO on UltraFeedback-LM. For a fair comparison, NS-DPO gets the same backbones (Llama-3.2-1B and Llama-2-7B, as in that paper), the same data splits and the same number of training runs as our method. Its $\gamma$ is tuned on a held-out slice from the most recent period. This is the realistic oracle-free protocol, and it is the most favourable non-oracle setting for NS-DPO. We additionally report NS-DPO with $\gamma$ chosen by peeking at the test-time preferences, as an upper bound for discounting. We count as beating NS-DPO only if our method matches it within noise on gradual drift and exceeds it by a margin larger than the seed-to-seed spread on abrupt and multi-switch drift.

The windowed DPO variant evaluated in that paper is also a baseline, with its window length tuned the same way. Two further baselines isolate where the gain comes from. The first replaces our likelihood-ratio test with a generic monitor: a CUSUM detector on the rolling held-out accuracy of a stationary reward head, which triggers a reset when accuracy falls. This tests whether the benefit comes from the structured time-varying model or simply from detecting changes at all. The second keeps our segmentation but uses hard weights, current segment only, instead of agreement-based soft weights. This separates the contribution of the segmentation from that of the per-segment weighting.

Why not just…?

Why not just tune $\gamma$ on recent validation data? This is the most serious threat, and it is the main NS-DPO baseline above. A single $\gamma$ has to pick one forgetting rate for the whole log. Under a stream that switches abruptly at irregular intervals, any one rate either forgets too slowly after a switch or throws away valid data during long stable stretches. Our multi-switch streams are built to expose exactly this. If tuned $\gamma$ matches us there as well, the claim fails and we will report that.

Why not just retrain on a sliding window? NS-DPO already outperformed its windowed DPO baseline. A window also discards stable data in stationary periods, whereas both NS-DPO and our merge-when-not-rejected rule can, in principle, keep the whole log when nothing changes. We include windowing with a tuned length, and expect it to lose most clearly on the stationary control.

Why not just use an off-the-shelf change-point detector on reward-model accuracy? This is plausibly almost as good as our method, and it is cheaper. Its weakness is that falling accuracy confounds drift with noisy or hard batches. It also gives no basis for partially reusing old segments. The CUSUM ablation decides this directly. If CUSUM ties our method, the right conclusion is that the speech-derived test adds nothing beyond detection, and we would simplify the method accordingly.

Why not just condition the reward or policy on the timestamp? A model given time as an input can only represent times it has seen. It cannot extrapolate to the present without the same unstated assumptions about how preferences evolve. We include a timestamp-conditioned DPO baseline on the text benchmarks for completeness.

Proposed Experiments

On the text side we use the drift benchmarks from Right Now, Wrong Then: TV-HH, UltraFeedback-RM/LM and 2C NSGO, which allow drift to be scheduled as sudden, gradual or absent. To these we add one new schedule: multiple switches at random times with long stable stretches between them. The metrics are reward accuracy on present-time preferences and, for UltraFeedback-LM, the AlpacaEval 2.0 length-controlled win rate.

Three checks decide the result. In the stationary control, our method must not fall below pooled DPO, since a merge rule that never rejects the null should recover it. On multi-switch drift, it must beat tuned-$\gamma$ NS-DPO beyond seed variance. Detected change points must line up with the scheduled switches, scored by detection delay and false alarms.

The trade-off analysis sweeps $q$ and the projection dimension $d$ against drift speed, mirroring the order-versus-basis-size trade-off reported for the speech tests. It reports the smallest switch the test can detect as a function of segment size.

The speech transfer builds semi-synthetic drifting streams from two sources. The first is the BVCC pairwise data used by Universal Preference-Score-based Pairwise Speech Quality Assessment, which reports 0.828 preference accuracy in its labels-available setting with a fixed preference function. The second is the SDiaReward-Dataset from Modeling and Benchmarking Spoken Dialogue Rewards with Modality and Colloquialness, whose reward model reaches 96.70% micro accuracy on its benchmark ESDR-Bench under a stationary judge. In these streams, the labeling criterion switches over time: for example, from human-rated naturalness to a preference for slower speech, mimicking a fatigued or noisy listener. We train lightweight pairwise speech reward heads on frozen WavLM features with the same set of weighting schemes. The test is whether adaptive segmentation tracks the switches as it does in text, and whether accuracy on the current criterion returns towards the stationary-judge level after each switch.

Because the criterion switches are constructed, this measures tracking ability, not real listener drift. Collecting timestamped spoken preferences from returning users is the natural next step, and the honest limitation of this proposal.