When people's tastes change, today's systems mostly respond by trusting old feedback less. A preference-tuned language model gives each past comparison an exponentially shrinking weight, and a recommender throws away its clusters every few hundred steps and starts over. Both treat old data as stale. That is wasteful, because old feedback still says a great deal about what makes a response or an item good in general. What changed is how much this particular user cared about each quality at that time. This note proposes a reward model that keeps all of its history. Each user's preferences are a hidden vector that wanders over time. The model alternates between two steps: it reconstructs where each user's preference vector was at every moment, and then it retrains the shared reward features using every past comparison, each read against the user's preference at the time it was made.
The core mechanism is iterative parameter estimation by Expectation-Maximization (EM), which alternates between inferring hidden quantities given current parameters and refitting parameters given those inferences. Here the hidden quantity is a per-user preference trajectory, inferred with a Kalman smoother (a recursive estimator that uses both past and future observations to locate a drifting hidden state). The parameters are the shared reward network and the speed at which each preference dimension drifts. Learning those drift speeds from data replaces the single hand-tuned forgetting rate that current methods depend on. Because the drift speeds can differ across dimensions, they can also pin down which preference axes are real. The claim is testable, and its main risk is clear: a simpler model with learned per-dimension discount rates may do just as well. The experiments are built to settle that question.
Core Claim
Let $\phi_\theta(x,y)\in\mathbb{R}^d$ be a shared feature map that scores response or item $y$ for context $x$. Let $z_{u,t}\in\mathbb{R}^d$ be user $u$'s hidden preference weights at time $t$. The model is
$$ z_{u,t}=z_{u,t-1}+\epsilon_{u,t},\quad \epsilon_{u,t}\sim\mathcal{N}(0,\mathrm{diag}(q)\,\Delta t),\qquad P(y_w\succ y_l\mid x,t)=\sigma\!\big(z_{u,t}^\top[\phi_\theta(x,y_w)-\phi_\theta(x,y_l)]\big), $$
Here $q\in\mathbb{R}^d_{\ge 0}$ is a per-dimension drift rate, $\Delta t$ is the time elapsed since the user's previous observation, and $\sigma$ is the logistic function, so the second equation is a Bradley-Terry comparison. EM alternates between two steps. In the E-step, a Laplace-approximated Kalman smoother computes the posterior mean and covariance of every $z_{u,t}$. In the M-step, $\theta$ takes gradient steps on the expected comparison log-likelihood under those posteriors, and $q$ is updated in closed form from the smoothed second moments.
We make three claims, in decreasing order of confidence. The first concerns data use: old comparisons keep contributing full information to $\theta$. They are explained by the old state of $z$ instead of being discounted, so the shared reward features should be learned more efficiently under drift than with recency weighting. The second concerns adaptive forgetting. The effective forgetting rate falls out of $q$ and the observation noise, and it automatically differs across dimensions and across users with sparse or irregular feedback. The third concerns identifiability. With a scalar drift rate, $z$ and $\phi$ can be rotated together without changing the likelihood. Heterogeneous diagonal drift rates break that rotation, in the same way that regime-dependent variances disentangle latent axes in End-to-End Identifiable and Consistent Recurrent Switching Dynamical Systems.
Relation to Prior Work
Three methods motivate this proposal by showing the gap. Right Now, Wrong Then: Non-Stationary Direct Preference Optimization under Preference Drift (NS-DPO) discounts each preference by $\gamma^{T-t_i-1}$ inside the Direct Preference Optimization (DPO) loss. It proves regret bounds, but it uses one global $\gamma$ tied to an unknown variation budget and applies it uniformly to all users and dimensions. Learning User Preferences in Non-Stationary Environments restarts user clusters in batches and evicts users whose tastes drift. That stops one user's drift from contaminating recommendations for others, at the cost of discarding history. Beyond Markovian Drifts: Action-Biased Geometric Walks with Memory for Personalized Summarization shows that structured memory beats simple decay, but its decay kernels are fixed by hand. Our model estimates the drift rate jointly with the reward instead of fixing it. Unlike NS-DPO it is per-user and per-dimension, and unlike Walk2Pers its memory dynamics are learned. We do not model Walk2Pers's suppression of skipped topics. That is a real omission for click and skip data.
Two papers supply the method. A state-space model of cross-region dynamic connectivity in MEG/EEG runs Kalman smoothing in the E-step and parameter updates in the M-step. It shows that joint estimation removes the spurious effects that decoupled two-step pipelines create. We reuse that structure with a pairwise-comparison observation model in place of a Gaussian sensor model. A Shared Low-Rank Adaptation Approach to Personalized RLHF shares one projection across users and learns a per-user component. Our $\phi_\theta$ and $z_u$ follow the same split, except that $z_u$ moves over time.
End-to-End Identifiable and Consistent Recurrent Switching Dynamical Systems both supplies and constrains the identifiability argument. Its guarantees require correct specification and well-separated regimes. We take only the idea that unequal variances identify axes, and we leave discrete regime switching to future work.
Several papers we cite only as competitors or evaluation sources. CARE: Adaptive Calibration for Reliable Recommendations recalibrates a black-box recommender after detecting a change point. It sits on top of a model rather than changing its internal representation, so it complements this work. AcCoRD shows that prompted frontier models fail on preferences that only emerge during interaction. We use it only as evidence against the prompting alternative, not as a target here.
Research Program
We will build the model on top of an off-the-shelf reward-model backbone. The last layer becomes $\phi_\theta$, and each user (or, for crowd data, each population) gets a latent trajectory $z_{u,t}$.
The E-step is an iterated extended Kalman smoother. It linearizes the log-sigmoid around the current mean, which is standard for logistic observation models. The M-step for $\theta$ uses the probit approximation to the expectation of $\log\sigma$ under the Gaussian posterior. This makes the uncertainty in $z$ shrink the effective learning signal, rather than plugging in only the posterior mean. The M-step for $q$ is the usual random-walk variance update, averaged over users, with an optional per-user scale under a weak prior.
To scale, we will run per-user smoothers in parallel, since each user's trajectory has only $d$ dimensions (small, for example 8 to 32), and amortize the expensive $\phi_\theta$ updates over minibatches between E-steps.
The analysis has three parts. The first is recovery of known drift rates on synthetic data. The second is how the advantage over discounting scales with the ratio of shared-structure learning to drift. Our working prediction is that the gain is largest when each user has few comparisons, so that discarding old data is most costly. The third is whether the learned axes stay stable across random seeds when the drift rates differ, and become unstable when they are equal, as the identifiability argument predicts.
Baselines
The naive reference point is a static personalized reward model with no time information, P-ShareLoRA-style (A Shared Low-Rank Adaptation Approach to Personalized RLHF). It is trained on all data with uniform weights and uses the same backbone and feature dimension. On Reddit TL;DR with five annotators that paper reports 74.65% preference accuracy for Llama-3 8B, which is the bar a stationary control must reach.
The strongest existing method on drift is NS-DPO in its reward-model form. On UltraFeedback-RM with change points it reports that standard DPO reward accuracy falls to about 50–55% under the shift while NS-DPO stays above 60–70%, with a gradual-drift advantage above 10 points (Right Now, Wrong Then). We will run NS-DPO on identical data splits, with $\gamma$ tuned on the same validation set using the same number of trials as our hyperparameters. Beating it means higher accuracy on future-time held-out comparisons across all three runs, with non-overlapping bootstrap intervals.
Because a single scalar discount is easy to beat, we add a stronger version of the obvious extension: a learned discount model. Each user and each feature dimension gets its own discount rate, fit by gradient descent on validation likelihood. Its compute budget matches ours in wall-clock time.
For recommendation we compare against COLLABORATIVE (Learning User Preferences in Non-Stationary Environments). We use its MovieLens 25m genre-switching protocol and tune its batch size $\Delta_T$, which it reports performs best at 100. That paper reports cumulative reward curves but no headline accuracy figure in our file, so the comparison is made by rerunning it.
Two ablations isolate the key ingredients. The first replaces the smoother with a forward-only filter, which tests whether reinterpreting the past matters or only forgetting does. The second fixes $q$ to a tuned scalar, which tests whether learning heterogeneous drift matters.
Why not just…?
Why not just learn a separate discount rate per user and per dimension? This is the most serious threat. A Kalman filter with random-walk dynamics settles into exponential forgetting, so in long, densely observed streams a well-tuned learned discount should match our filter. Where they should differ is in what the shared reward network learns. A discounted model trains $\theta$ mostly on recent data, while the smoother lets every old comparison train $\theta$ at full strength once it is placed against the right past preference. A second difference is irregular or sparse feedback, where the filter's gain adapts automatically. The smoother-versus-filter ablation and the learned-discount baseline settle this directly. If the learned discount matches us on held-out accuracy in the low-data regime, we will report that the contribution reduces to interpretable drift estimates and drop the efficiency claim.
Why not just give the reward model a timestamp and a user embedding as inputs? A time-conditioned network can in principle represent drift. However, it must learn the drift process from scratch for each user, and it has no built-in notion that preferences change continuously, so it should overfit when users have few comparisons. We include it as a baseline with matched parameter count. If it wins on dense data but loses on sparse users, that supports the structural prior. If it wins everywhere, the prior is not earning its keep.
Why not just prompt a long-context model with the user's history? Two cited results argue against this. On AcCoRD, prompted frontier models stay below a 30% perfect-outcome rate and fall to 0.09 on emergent preferences (AcCoRD). On personalized summarization, a 170M-parameter structured model beats prompted 7–14B models by 0.22 PerSEval (Beyond Markovian Drifts). We will still run one prompted baseline on the synthetic drift task to confirm this on our data.
Why not just retrain periodically on recent windows? Short windows are what COLLABORATIVE needs to avoid cross-user contamination, and they throw away history. Our model handles contamination by giving each user a separate state, and it keeps all the data. The MovieLens comparison tests whether this pays off.
Proposed Experiments
The first experiment is a controlled drift study, and it is decisive. We will build simulated users whose hidden weights over a fixed set of reward-attribute features follow random walks with known drift rates. Rates will differ across dimensions and users, and some users will be stationary. Labels come from Bradley-Terry sampling on UltraFeedback-style prompt and response pairs. We will measure three things: recovery of $q$, accuracy on comparisons from the future, and accuracy as each user's number of comparisons falls. The main prediction is that the smoother beats the learned-discount baseline most clearly at low per-user counts. If it does not, the efficiency claim fails.
The second experiment uses the published drift benchmarks UltraFeedback-RM and TV-HH, with NS-DPO's change-point protocol. The target is NS-DPO's reported reward accuracy of above 60–70% under shift. No reported result was found on Lacuna for TV-HH, so we will rerun NS-DPO there.
The third experiment is a stationarity control on Reddit TL;DR with five annotators. The learned $q$ should come out near zero, and accuracy should stay within noise of 74.65%. This checks that the method does no harm when there is no drift.
The fourth experiment tests real, slow drift on the New Yorker Caption Contest (yguooo/newyorker_caption_ranking). That dataset covers 365 weekly contests over eight years with more than 250 million ratings (Humor in AI). We treat the rater population as one drifting user and predict next-contest caption ratings from ratings on earlier contests. No reported result was found on Lacuna for this temporal prediction task, so the baselines are static and discounted models we train ourselves. The interesting outputs are whether $q$ is significantly nonzero and which learned humor axes drift.
The fifth experiment is MovieLens 25m with genre switching, against COLLABORATIVE, using held-out next-window ranking accuracy.
Across all experiments, the identifiability check reruns training with five seeds. It compares axis alignment, measured by absolute correlation after optimal permutation, between heterogeneous-drift and equal-drift conditions.
References
- End-to-End Identifiable and Consistent Recurrent Switching Dynamical Systems
- Right Now, Wrong Then: Non-Stationary Direct Preference Optimization under Preference Drift
- Learning User Preferences in Non-Stationary Environments
- Beyond Markovian Drifts: Action-Biased Geometric Walks with Memory for Personalized Summarization
- A state-space model of cross-region dynamic connectivity in MEG/EEG
- A Shared Low-Rank Adaptation Approach to Personalized RLHF
- CARE: Adaptive Calibration for Reliable Recommendations
- AcCoRD: Evaluating User-Agent Collaboration Under Realistic User Preference Dynamics
- Humor in AI: Massive Scale Crowd-Sourced Preferences and Benchmarks for Cartoon Captioning