People's preferences change over time. A reward model, meaning the model that scores which of two responses a person would prefer, is usually trained as if every past comparison came from one fixed person. One principled fix is to treat the preference weights as a hidden state that moves over time and to track it with a Kalman filter, a recursive Bayesian estimator that maintains both a best guess and a covariance matrix describing its uncertainty. When the preference state lives in the feature space of a language model, with $d$ in the thousands, and there is one such state per user, a full $d \times d$ covariance per user becomes unaffordable. The standard escape is to keep only a rank-$r$ approximation of the covariance. That approximation has a known flaw: it throws away uncertainty in every direction outside the kept subspace. A drifting-preference tracker cannot afford this flaw, because the filter becomes certain about exactly the directions in which the person has not yet been observed to change. When the person does change along one of those directions, the filter barely reacts.
This note proposes a filter for pairwise preference feedback whose covariance is a low-rank matrix plus a diagonal matrix. Whenever the low-rank part is truncated, the variance it discards is folded back into the diagonal, so every coordinate's marginal uncertainty is preserved exactly. The process-noise scale is re-estimated online from how surprised the filter is by incoming comparisons. The claim is that this recovers from preference flips in unexplored directions about as fast as an exact full-covariance filter, at $O(dr)$ memory per tracked state. Pure low-rank filters and scalar time-discounting both fail on such flips, and the proposal is to show that on non-stationary preference benchmarks and eight years of crowd humor ratings.
Core Claim
Existing work splits along two lines. Dynamic Matrix Factorization: A State Space Approach tracks user preference vectors with a Kalman filter and smoother. It shows that this follows curved preference trajectories where linear-drift models such as TimeSVD overfit, but it assumes Gaussian rating observations and stays at $d \le 10$. Right Now, Wrong Then (NS-DPO) keeps the correct pairwise model, a time-indexed Bradley-Terry likelihood. It replaces state tracking with a single discount $\gamma^{T-t_i}$ on older samples, so every latent direction forgets at the same rate and there is no posterior uncertainty.
On the scaling side, The Rank-Reduced Kalman Filter (RRKF) cuts the cost of covariance propagation to $O(nr^2)$, for state dimension $n$ and kept rank $r$. It is derived only for linear-Gaussian observations, and its authors name overconfidence from the discarded spectral tail as its main limitation. That limitation motivates this proposal directly. Probabilistic ODE Solutions in Millions of Dimensions supplies two pieces of the method: diagonal covariance structure that scales linearly with dimension, and online calibration of the diffusion scale.
The new object is a filter with three properties together: it handles a Bradley-Terry likelihood, it uses a low-rank-plus-diagonal covariance, and truncation moves discarded variance into the diagonal rather than deleting it. The main empirical prediction is specific. On preference flips aligned with directions outside the current subspace, pure truncation should recover slowly or not at all, while the proposed filter should track the exact filter. NS-DPO-style discounting should be competitive on global flips but lose when only a subset of preference directions drift.
Research Program
The model is kept deliberately simple. Each response $a$ is embedded by a frozen reward-model or encoder feature map $\phi(a) \in \mathbb{R}^d$, and the preference weights $w_t$ follow a random walk. A comparison between responses $a$ and $b$ at time $t$ has likelihood
$$ w_t = w_{t-1} + \eta_t,\quad \eta_t \sim \mathcal{N}(0, Q_t), \qquad P(a \succ b \mid w_t) = \sigma\big(w_t^\top z\big),\; z = \phi(a) - \phi(b), $$
where $\sigma$ is the logistic function and $Q_t$ is the process-noise covariance. The posterior is approximated as Gaussian with covariance
$$ \Sigma_t \approx U_t U_t^\top + D_t, $$
where $U_t \in \mathbb{R}^{d \times r}$ is the low-rank factor and $D_t$ is a nonnegative diagonal matrix.
Each comparison triggers an assumed-density or Laplace update, following the non-Gaussian filtering primitives in State Space Gaussian Processes with Non-Gaussian Likelihood. This is a rank-one correction whose direction is $\Sigma_t z = U_t(U_t^\top z) + D_t z$, computable in $O(dr)$ time. The correction raises the rank of the low-rank part to $r+1$. We re-truncate with a thin QR decomposition and a small eigendecomposition of size $(r+1)\times(r+1)$, at total cost $O(dr^2)$. The dropped eigenpair $(\lambda, v)$ is added to the diagonal as $\lambda\,\mathrm{diag}(v v^\top)$. This diagonal-preserving truncation leaves every coordinate's marginal variance equal to its pre-truncation value, so the filter cannot become overconfident coordinate by coordinate.
The process noise is split as $Q_t = q^U_t\,U_t U_t^\top/\mathrm{tr}(U_tU_t^\top) + q^D_t I$, with one scale for drift inside the tracked subspace and one for drift outside it. Following the diffusion calibration idea of Krämer et al., we set the two scales online by maximizing the predictive log-likelihood of a recent window of comparisons. Sustained surprise then inflates exactly the channel that explains it.
For many users, each user's weights are $w_{u,t} = w_t + \delta_{u,t}$: a shared state plus a personal offset. The shared state gets rank $r$. Each offset gets a small rank $r_u$ within the shared basis, plus its own diagonal. This mirrors the shared up-projection of A Shared Low-Rank Adaptation Approach to Personalized RLHF (P-ShareLoRA), which learns a shared low-rank factor and per-user factors but has no dynamics. The cross-covariance between the shared and personal blocks is dropped, a block mean-field approximation, and we measure what this costs against the exact joint filter on small user counts. Memory is $O(Mdr_u)$ for $M$ users, compared with $O(Md^2)$ for full covariances.
Baselines
The naive reference is a stationary Bayesian logistic reward head. It is fit on all comparisons with equal weight, on the same frozen features, with the same prior. It stands in for standard DPO's stationarity assumption, which Right Now, Wrong Then reports collapsing to about 50–55% reward accuracy after a mid-dataset preference flip. Beating it is necessary but not interesting.
The strongest competitor is exponential discounting in the style of NS-DPO. It is the strongest because it costs nothing, has regret guarantees, and is reported to hold reward accuracy above 60–70% under the same flip, as well as improving AlpacaEval 2.0 length-controlled win rate from 7.29 to 8.93 on UltraFeedback-LM (Xu et al.). We run it as a discount-weighted maximum-likelihood head on the identical features and data stream. Its $\gamma$ is tuned by the same held-out search budget we give our two noise scales. We also compare against the published numbers on the UltraFeedback-RM flip setting. Beating it means higher time-averaged reward accuracy and lower log loss on subspace-drift streams, together with no loss on global flips and stationary streams.
The exact reference is the full-covariance Kalman filter with the same assumed-density update. For a single shared state at $d \approx 4096$ it is feasible, because a rank-one update is $O(d^2)$. The success criterion against it is closeness, measured by KL divergence and Frobenius distance of the posterior, rather than improvement over it.
The ablation isolating the key ingredient is the same filter with truncated variance simply discarded (RRKF-style), with and without the two-channel noise calibration. A diagonal-only filter in the style of Krämer et al.'s diagonal solver, which drops correlations, and an ensemble Kalman filter with tuned inflation at matched memory complete the set. The Rank-Reduced Kalman Filter found the ensemble filter less accurate than RRKF on London air quality, but the file gives no RMSE value, so no reported result is found on Lacuna for that comparison. For personalization accuracy, the static reference is P-ShareLoRA's 74.65% preference prediction accuracy on Reddit TL;DR (top five annotators, Llama-3 8B, Zhong et al.). That model fine-tunes the network, so we report it as context rather than as a matched comparison.
Why not just…?
Why not just discount old data, as NS-DPO does? This is the real threat. On a global preference reversal, where every direction flips together, a well-tuned $\gamma$ may match any filter. The proposal's advantage should appear only when drift is confined to a few directions. There, one scalar discount must either forget stable preferences too fast or adapt to the drifting ones too slowly. The subspace-drift experiments below are designed to settle this. If discounting matches us on those as well, the Bayesian machinery is buying only calibrated uncertainty, and we will say so.
Why not just keep the full covariance, since each comparison is only a rank-one update? For one shared state this is correct: at $d = 4096$ the covariance is about 64 MB and each update is cheap. That is why we use the full filter as the gold reference rather than as a strawman. The cubic and quadratic costs bite in three places. Per-user states make memory grow as $Md^2$. Learning structured process noise or transitions by expectation-maximization requires smoothing with $d \times d$ products. Scoring many candidate queries by expected information gain costs $O(Kd^2)$ for $K$ candidates. The scaling experiment reports the user count at which full covariance stops fitting on a single accelerator.
Why not just drop correlations and use a diagonal covariance? Diagonal filters scale linearly, as Krämer et al. show for ODE solving. In learned feature spaces, however, preference directions are rarely axis-aligned. A diagonal posterior misjudges how information about one comparison should move weights on correlated features. The diagonal-only baseline measures how much this costs.
Why not just fine-tune the reward model online with fast and slow adapters? Online DPO with Fast-Slow Chasing reports gains of about 8% and 5% in win rate over DPO and a Score Forgetting Ratio of −1.418 on TL;DR. That method adapts the whole network, which our frozen-feature filter does not, so it may win when the feature space itself shifts. It has no posterior, however, and cannot report which preferences it is unsure about. We include it on TL;DR at matched update counts and treat a loss there as evidence that feature drift, not weight drift, dominates.
Proposed Experiments
The first decisive check is a controlled orthogonal-flip test on UltraFeedback-RM features. After a burn-in period, we reverse the sign of the true weights along a direction chosen orthogonal to the filter's current top-$r$ subspace. We record reward accuracy against the number of comparisons since the flip. The prediction is that pure truncation plateaus, the proposed filter recovers within a small constant factor of the full filter, and discounting recovers but degrades accuracy on the unflipped directions.
Second, we run the TV-HH and non-stationary GlobalOpinionsQA streams from Xu et al. The file reports no numeric results for these, so we reproduce the NS-DPO head ourselves. Metrics are time-averaged reward accuracy, log loss, and expected calibration error, plus the KL divergence of each approximation to the full filter as the rank $r$ varies.
Third, for real drift without synthetic flips, we use the New Yorker caption ranking data from Humor in AI. It contains 250 million crowd ratings across 365 contests over eight years. Pairs are formed within each contest from rating differences, and each contest is predicted using only earlier contests. No reported result is found on Lacuna for temporal tracking on this dataset, so every baseline is run fresh. The question is whether the calibrated residual channel detects shifts in crowd taste that a fixed $\gamma$ misses.
Fourth, per-annotator Reddit TL;DR tests the multi-user variant. Annotation order is used if the release preserves it; otherwise we inject per-user drift. We report accuracy together with memory and time against user count.
The proposal fails if diagonal-preserving truncation does not beat plain truncation on the orthogonal-flip test. It is merely redundant if tuned discounting matches it on subspace drift and calibration.
References
- Dynamic matrix factorization: A state space approach
- Right Now, Wrong Then: Non-Stationary Direct Preference Optimization under Preference Drift
- The Rank-Reduced Kalman Filter: Approximate Dynamical-Low-Rank Filtering In High Dimensions
- Probabilistic ODE Solutions in Millions of Dimensions
- State Space Gaussian Processes with Non-Gaussian Likelihood
- A Shared Low-Rank Adaptation Approach to Personalized RLHF
- Online DPO: Online Direct Preference Optimization with Fast-Slow Chasing
- Humor in AI: Massive Scale Crowd-Sourced Preferences and Benchmarks for Cartoon Captioning