Telling Real Preference Drift Apart from Who Stopped Showing Up

Systems that learn from human feedback increasingly assume that people's preferences change over time, and they respond by trusting recent feedback more than old feedback. A single hidden assumption sits underneath that design. The feedback at each moment is treated as a fair sample of the people the system serves. Digital platforms break this assumption in a predictable way. Users who hit interface friction, lack fluency, or get frustrated leave quietly, and research on digital selection bias shows that the ones who leave are disproportionately older, less educated, and less connected. When they leave, the average preference in the feedback stream moves even though no individual has changed their mind. A method that discounts old data reads this shift as "drift" and follows it, which pushes the model further toward the users who remained.

This note proposes to separate the two sources of change. The first is genuine within-person drift. The second is change in the mix of who is contributing. The separation uses the repeated-observation structure that interaction logs already have, and the proposal then builds a preference-optimization objective that targets an explicitly chosen population at the current time, not whoever happens to still be talking. The technical ingredients are borrowed from survival analysis (reweighting remaining participants by their estimated chance of staying) and from time-discounted preference optimization. The contribution is to combine them. Alongside the method we contribute a benchmark on which the two sources of change produce identical aggregate curves, so that methods that only look at the aggregate cannot tell them apart.

Core Claim

Right Now, Wrong Then (NS-DPO) replaces the static preference model behind Direct Preference Optimization with a time-varying one. It weights a preference collected at time $t_i$ by $\gamma^{T-t_i-1}$, where $T$ is the current time and $\gamma\in(0,1]$ is a discount. Under flips, it keeps reward accuracy above 60–70% where standard DPO falls to about 50–55%. Its drift benchmarks (TV-HH, UltraFeedback with change points, and 2C NSGO, which moves GlobalOpinionsQA labels from US-centric to Japan-centric) relabel a fixed stream. None of them model who participates. Our claim is that whenever dropout is correlated with preference, time discounting estimates the preferences of the survivors rather than of the population. Discounting makes this worse, because it concentrates weight on the late, most filtered data.

The observed change decomposes cleanly. Let $w_t(u)$ be user $u$'s share of feedback at time $t$ and $p_t^u$ that user's preference probability for a given comparison. Then the change in aggregate preference $\bar p_t=\sum_u w_t(u)p_t^u$ satisfies

$$ \bar p_T-\bar p_0=\underbrace{\sum_u w_0(u)\,(p_T^u-p_0^u)}_{\text{within-user drift}}+\underbrace{\sum_u (w_T(u)-w_0(u))\,p_T^u}_{\text{composition shift}}. $$

Only the first term reflects anyone changing their mind. Logs identify users across sessions, so the first term can be estimated from users observed at several times. Leavers stop producing data, so their $p_T^u$ is unobserved. We recover it by reweighting stayers who resemble them, a standard trick from survival analysis called inverse probability of censoring weighting. The training weight for pair $i$ from user $u_i$ becomes

$$ \omega_i=\gamma^{\,T-t_i-1}\cdot\frac{1}{\hat\pi(u_i,t_i)}, $$

where $\hat\pi(u_i,t_i)$ is the estimated probability that a user with $u_i$'s interaction history up to $t_i$ is still contributing at $t_i$. It is fit on user identity, past preferences, and in-session dissatisfaction signals, and it needs no demographic labels. Because $\gamma$ now acts only on within-user evidence, its tuning should also become more stable. This matters because NS-DPO's optimal $\gamma$ depends on a variation budget that is unknown in deployment.

The approach carries an honest limitation. If people leave because their own preferences drifted in ways their history does not reveal, the missing data are "not at random," and no reweighting identifies the target. We therefore pair the estimator with a sensitivity analysis that reports how far conclusions move under a bounded amount of such dependence.

Research Program

The work has three parts. The first is a diagnostic. It estimates the decomposition above on any timestamped, user-identified preference log and reports what fraction of apparent drift is composition. It is cheap and useful even without retraining.

The second is the training objective above, built as a drop-in change to NS-DPO. It has an explicit choice of target population. The choices are the users present at launch, a stated reference population, or current survivors, and choosing current survivors recovers plain NS-DPO. Making the target explicit is itself a contribution. Current methods pick "survivors" implicitly, and Partnering with a global platform shows how badly survivor-style samples miss offline and low-connectivity populations.

The third is a benchmark in which drift and attrition are controlled separately, described under experiments.

Prior work plays three roles here. Two papers motivate the problem. A First Look at Selection Bias in Preference Elicitation shows that users choose what to rate and that inverse propensity scoring corrects much of the error: on Yahoo! R3, MAE falls from 1.3041 to 0.8327. Its propensities and preferences are static, however, whereas ours vary in time and are tied to exit. DRIFT learns from users' corrections and objections. It rewards exactly the articulate users whom Measuring Non-Expert Comprehension of Fairness Metrics and the older-adults think-aloud study suggest are better able to engage. We treat its dissatisfaction signals as predictors of who leaves, rather than only as training pairs.

Kreuter et al. constrain the problem. They found that trends over time are more robust than point estimates because selection stays roughly constant. Our problem exists precisely when selection is not constant, so the diagnostic should report near-zero composition share in stable regimes, and we test that it does. The Vietnam web-sampling study calibrates how strong demographic filters can be (mean age 22, 87% post-secondary education). NS-DPO supplies the base objective.

Baselines

The naive reference is standard DPO on the pooled stream, with uniform weights over time and users. It shows how much any drift handling helps at all.

The strongest existing drift method is NS-DPO. Its reported results are an AlpacaEval 2.0 length-controlled win rate of 8.93 against DPO's 7.29 on UltraFeedback-LM, and reward accuracy above 60–70% versus about 50–55% for DPO after flips on TV-HH (both from Right Now, Wrong Then). We run it on identical data and model sizes, with the same number of $\gamma$ values searched as we give our own method, using a held-out validation slice drawn from the target population. We beat it if we achieve higher target-population reward accuracy under preference-correlated attrition while matching it (within seed noise) when attrition is random.

The strongest selection-bias method is static inverse propensity weighting in the style of MF-IPS from A First Look at Selection Bias, ported to DPO, with propensities fit once over the whole log. We give it the same propensity-model features and tuning budget.

The most dangerous competitor is stratified NS-DPO with post-stratification. It is given ground-truth group labels, reweights each period to the target's group mix, and discounts within group. In simulation it acts as an oracle for label-driven attrition.

A personalized reward model, P-ShareLoRA, is included for the TL;DR setting. It reports 74.65% preference accuracy on Reddit TL;DR with Llama-3 8B. We aggregate its per-user rewards with uniform and with survivor weights.

Ablations isolate the key ingredient. We drop the retention weights (leaving NS-DPO), drop the discount (leaving pure censoring weights), and fit $\hat\pi$ without dissatisfaction features. This last ablation tests whether in-session friction carries the signal about who leaves.

Why not just…?

Why not just use a sliding window or NS-DPO's discounting? Discounting cannot tell composition from drift, because both change the recent average in the same direction. Under attrition that is correlated with preference, a shorter window moves weight onto the most filtered users. The prediction is that NS-DPO gets worse as $\gamma$ shrinks on our composition-only condition. This would be the reverse of its behavior on the original benchmarks, where every period samples the same population. That condition is the direct test.

Why not just post-stratify by demographics each period? This is a real threat. If dropout depends only on recorded group membership and preferences are homogeneous within groups, post-stratified discounting identifies the target and should tie with our method. We expect it to tie in that condition, and we include it deliberately. Our method should win when dropout is driven by things that are not recorded, such as a latent user type, accumulated dissatisfaction, or differences within a group. Demographic labels miss these, and interaction history captures them. Platforms also often lack reliable demographics. If stratification matches us even with history-driven attrition, the contribution narrows to the diagnostic and the benchmark, and we will say so.

Why not just personalize per user, so population averages don't matter? Personalized reward models like P-ShareLoRA or CoPL still hold only a frozen, stale representation of leavers. A shared model or default policy still needs some way to weight users. CoPL's zero-shot aggregation for new users draws on neighbors that are themselves survivors. We test this by aggregating P-ShareLoRA rewards under survivor weights versus our weights.

Why not just ask users when uncertain? AcCoRD finds that clarification prompting leaves triggered-preference scores as low as 0.09 and makes weaker models over-ask. Asking more also adds exactly the friction that drives low-fluency users away, so it cannot recover people who have already left.

Proposed Experiments

The central benchmark extends 2C NSGO, built on GlobalOpinionsQA (Towards Measuring Subjective Global Opinions). Simulated respondents carry persistent identities and country-conditioned opinions. We construct three regimes that produce the same aggregate US-to-Japan curve. In the first, individuals genuinely change their opinions. In the second, US-leaning respondents leave. In the third, both happen. No numeric result for 2C NSGO is reported on Lacuna. The decisive check is whether each method recovers the true opinion distribution of the full target population at $T$, including leavers whose simulated current opinions are known, and whether the diagnostic's composition share matches the true share in each regime. Any method that sees only the aggregate should score the same in all three regimes. Ours should not.

The second benchmark uses per-annotator Reddit TL;DR preferences as distinct user types. Attrition follows either observed type or accumulated dissatisfaction with the current model's summaries. Here attrition feeds back on the model: a model that serves some users poorly loses them and then sees even less of their preferences. We also run UltraFeedback-P and PersonalLLM user profiles. These benchmarks have no reported per-user numbers on Lacuna, so their CoPL baseline gives a ranking only.

To calibrate how strong the missing-not-at-random selection should be, we use Yahoo! R3 and Coat, which pair user-chosen ratings with randomly assigned ones. This sets how aggressively attrition should track preference in the simulations, rather than choosing it arbitrarily.

The metrics are reward accuracy on held-out target-population pairs and per-group accuracy, with emphasis on the worst group. We also measure error in the estimated composition share, AlpacaEval 2.0 length-controlled win rate on the UltraFeedback-LM variant for comparability with NS-DPO, and the spread of conclusions across the sensitivity-analysis bounds.

The proposal fails in two cases. The first is if stratified NS-DPO ties with our method under history-driven attrition. The second is if the retention weights raise variance enough to erase gains at realistic log sizes. In that second case we would report the diagnostic as the main deliverable.