Telling Real Preference Drift from Noisy Feedback with Conformal Tests

When people's preferences change over time, the systems trained on their feedback have to decide which old feedback to keep trusting. The leading fix for language-model alignment, Right Now, Wrong Then: Non-Stationary Direct Preference Optimization under Preference Drift (NS-DPO), down-weights old comparisons by a fixed exponential decay. Choosing that decay rate requires knowing in advance how much preferences will change. The decay also treats every comparison from a given time the same way, so one careless annotator's flipped label counts exactly as much as the first sign of a real shift.

We propose to replace the hand-tuned decay with a statistical test that has a guaranteed false-alarm rate. Each new preference comparison receives a conformal p-value, a distribution-free score of how surprising that comparison is relative to recent held-out feedback. A running wager on these p-values (a test martingale) grows only when surprises persist. Isolated surprising comparisons are down-weighted as likely noise. Persistent surprise is declared drift, and the training data is cut at the estimated change point. In two sentences: conformal anomaly detection already gives calibrated false-alarm control, but it has only been applied to sensor signals and event timings. We apply it to streams of human preference comparisons, so that the alignment procedure can tell a real change in what people want from noise, with a guaranteed bound on how often it raises a false drift alarm.

Core Claim

The claim is that a two-level conformal monitor on preference comparisons does two things at once. It matches or beats NS-DPO tuned with an oracle-chosen decay on drifting benchmarks. It also does not degrade when the stream has no drift but contains outlier labels, which is the condition under which any fixed decay throws away good data. The new ingredient is not "conformal plus preferences" in general. It is the separation of two failure modes that the current literature conflates. The first is a single non-conforming comparison, which a per-sample p-value flags. The second is a sustained loss of exchangeability, which a martingale flags with a bound that holds over the whole stream. "Exchangeability" here means, informally, that the order of the data carries no information.

The file states this tension directly. Conformal methods treat non-exchangeability as an assumption violation to be repaired. For example, Adaptive Conformal Anomaly Detection with Time Series Foundation Models for Signal Monitoring. (W₁-ACAS) learns weights over past calibration points so that p-values stay uniform under drift. Preference-drift work, by contrast, treats non-stationarity as the very signal it needs. We side with the second view. The loss of exchangeability is what we want to detect, not something to calibrate away.

How It Differs From the Closest Work

Three conformal anomaly detectors supply the method but target different data. Conformal Anomaly Detection in Event Sequences (CADES) controls false-positive rate on continuous-time event logs using arrival-time and interval scores. Its authors leave high-dimensional "marks" as future work, and it cannot tell a benign behavioural change from a fault. W₁-ACAS reweights calibration residuals from frozen forecasters on numeric signals. Context-Aware Online Conformal Anomaly Detection with Prediction-Powered Data Acquisition controls online false discovery rate but assumes known operating contexts. None of the three scores discrete pairwise choices, and none feeds its detection back into training.

On the recommendation side, Confidence-aware Fine-tuning of Sequential Recommendation Systems via Conformal Prediction uses conformal sets only as a training regularizer under an exchangeability assumption, and explicitly names non-exchangeable conformal methods as future work. Estimating and Penalizing Preference Shift in Recommender Systems models drift with a learned hidden Markov model but attaches no significance test to observed shifts.

NS-DPO is the method we aim to replace. It motivates the gap through its unknown "variation budget" and its uniform per-timestamp weights, and it is the strongest competitor. CADET: Calibrated Anomaly Detection for Mitigating Hardness Bias constrains our score design. It shows that raw non-conformity scores are not comparable across inputs of different intrinsic difficulty, which is why we rank scores against a calibration set rather than threshold them directly.

Research Program

The base model is a DPO-trained policy whose implicit reward $r_\theta$ defines a Bradley–Terry probability that the chosen response beats the rejected one. The non-conformity score of a comparison is its negative log-likelihood under this model, $S = -\log \sigma(r_\theta(y_w) - r_\theta(y_l))$. Here $\sigma$ is the logistic function, $y_w$ is the preferred response, and $y_l$ is the rejected one.

Since the last declared change point, the system holds out a rolling calibration set of $n$ comparisons that are never trained on. Each incoming comparison $t$ gets a randomized conformal p-value:

$$p_t = \frac{\#\{i : S_i > S_t\} + u_t\,(1 + \#\{i : S_i = S_t\})}{n+1}.$$

Here $S_i$ are the calibration scores and $u_t$ is uniform on $[0,1]$. The randomization breaks ties, which matter because many comparisons will receive near-saturated scores. If the preference-generating process has not changed since the last change point, $p_t$ is uniform, by the standard argument in the conformal anomaly detection literature.

Persistence is tracked with a conformal test martingale $M_t = \prod_{s \le t} f(p_s)$. The function $f$ is a betting function that integrates to one on $[0,1]$ and rewards small p-values; we use a mixture of power functions. Ville's inequality bounds the probability that $M_t$ ever exceeds $1/\delta$ under the null by $\delta$. This gives a stream-wide false drift-alarm guarantee, which neither NS-DPO nor the recommender drift models provide.

Detection then drives the training procedure. Below threshold, a comparison with $p_t \le \alpha_{\text{out}}$ is down-weighted in the DPO loss as a probable outlier. When $M_t$ crosses $1/\delta$, the change point is estimated as the start of the martingale's steepest sustained climb. Pre-change data is then dropped, or softly weighted in an ablation. The model is fine-tuned on post-change data, and calibration restarts.

The analysis has three parts. First, we prove that the false-alarm guarantee holds when training and calibration data come from the same stationary segment. Second, we study empirically how validity degrades when the reward model is updated online, since that update couples the scores to the stream. Third, we characterize detection delay as a function of drift magnitude. Delay should scale roughly inversely with the per-sample divergence between pre- and post-change score distributions, the standard rate for likelihood-ratio-style sequential tests.

Baselines

The do-nothing reference is standard DPO trained on the full stream. On preference-flip benchmarks, NS-DPO reports its reward accuracy falling to about 50–55%.

The strongest existing method is NS-DPO itself. We run it in two ways. The first chooses γ on a validation stream drawn from a different drift schedule, which is the realistic setting. The second chooses γ with oracle knowledge of the test change point, which gives NS-DPO an advantage we do not have. The reported numbers we must match or beat are reward accuracy "above 60–70%" under preference flips, a gain of over 10% in reward accuracy on gradual drift, and a length-controlled win rate of 8.93 versus 7.29 for DPO on UltraFeedback-LM, judged by GPT-4 on AlpacaEval 2.0, all from Right Now, Wrong Then. Beating it means matching oracle-tuned NS-DPO within noise on its own benchmarks and exceeding validation-tuned NS-DPO, while also winning clearly on the outlier-contaminated and no-drift streams described below.

A sliding-window DPO baseline has its window size tuned with the same budget as γ.

Two baselines isolate the conformal ingredient. The first is a CUSUM detector, the classical cumulative-sum change detector, run on the same negative log-likelihood scores. Its threshold is set by a Gaussian approximation or by simulation. This mirrors the Gaussian-threshold baseline that W₁-ACAS beat, 0.925 versus 0.714 point-adjusted F1 on the TiRex forecaster in Adaptive Conformal Anomaly Detection with Time Series Foundation Models for Signal Monitoring.. The second applies W₁-ACAS-style weighted calibration to our scores, which tests whether adapting the calibration to drift beats detecting it.

Two ablations remove one level of the monitor each. One drops the martingale and keeps only per-sample thresholding. The other drops outlier gating and keeps only the martingale.

All methods share the same backbone as the NS-DPO experiments, the same data order, the same number of gradient steps, and the same hyperparameter search budget. The held-out calibration comparisons are also given back to the baselines as training data, so our method gains no hidden data advantage.

Why not just…?

Why not just tune NS-DPO's decay rate on a validation set? This is the real threat. If drift schedules at deployment resemble validation schedules, a tuned γ may be hard to beat on accuracy alone. Our experiments settle this by evaluating on schedules unseen during tuning: sudden versus gradual drift, and multiple change points versus one. They also evaluate on streams with no drift, where any γ below one wastes data and our monitor should leave training unchanged. If validation-tuned NS-DPO matches us across these conditions, the remaining contribution is only the false-alarm guarantee, and we will report that honestly.

Why not just run CUSUM or a likelihood-ratio change detector on the reward-model loss? CUSUM has good detection delay, but its threshold assumes a known pre-change score distribution. That distribution moves whenever the reward model or the prompt mix changes. The CUSUM-on-same-scores baseline measures directly whether its empirical false-alarm rate on stationary streams exceeds the target, as the 3S statistic's did on LOGS in Conformal Anomaly Detection in Event Sequences. If CUSUM stays calibrated here, the conformal layer is unnecessary, and the experiment will show that.

Why not just use a sliding window? A window has the same unknown-variation-budget problem as γ, and it is equally blind to individual outlier labels. The contaminated-stream experiment, where flipped labels are injected at random times, is designed so that any fixed window must either keep the outliers or discard good data.

Why not just calibrate away the drift, as W₁-ACAS does? Reweighting calibration keeps p-values uniform, but it hides the fact that preferences changed. It therefore gives the training procedure no signal to cut stale data. We run this adapted baseline directly. If feeding its weights into DPO performs as well as our approach, the detection framing adds nothing.

Proposed Experiments

The primary benchmarks are TV-HH, UltraFeedback-RM, UltraFeedback-LM, and 2C NSGO from Right Now, Wrong Then. Their change points are known, which allows us to measure detection delay and false-alarm counts alongside post-change reward accuracy and length-controlled win rate.

Three further stream types sharpen the test. The first is unmodified HH-RLHF with no drift, which checks whether the empirical false-alarm rate stays at or below δ. The second injects 5–10% randomly flipped labels with no drift, where the decisive check is that our method keeps reward accuracy close to clean-data DPO while NS-DPO and windowing lose accuracy. The third combines a change point with injected outliers, where we should detect the change with bounded delay and without false alarms triggered by outlier bursts.

A secondary user-level study applies the same monitor to per-user sequential recommendation on the Amazon Review subsets used by Confidence-aware Fine-tuning. It scores next-item negative log-likelihood under SASRec, with synthetic category switches inserted at known times. No reported drift-detection result was found on Lacuna for these datasets, so this study is exploratory.

The idea fails if either of two things happens. It fails if oracle-free NS-DPO matches us on unseen schedules and contaminated streams. It also fails if online reward-model updates break the false-alarm guarantee badly enough in practice that it no longer holds approximately.