Forgetting Only What Changed: Low-Rank Directional Discounting for Preference Learning Under Drift

People's preferences change over time, and systems that learn from human comparisons, such as reward models, preference-tuned language models and recommenders, handle this by down-weighting old feedback. The standard recipe gives every example a weight that decays exponentially with its age. That single decay rate discards old evidence about everything, including the parts of a person's taste that never changed. The features these systems learn from are also strongly correlated with one another. With correlated features, the data pin down only a few combinations of weights well, and most other directions are barely constrained. Throwing data away then makes the fitted weights swing wildly along those poorly constrained directions. So the fix for drift makes the correlated-feature problem worse.

We propose to forget selectively. We estimate the low-dimensional subspace in which preferences are actually moving. Old feedback is discounted only along that subspace and kept at full weight everywhere else. The central claim is that preference drift and feature correlation live in different subspaces. Methods that discount uniformly, or that discount along the principal components of the features, confuse the two. A drift-subspace estimator should beat both exactly when drift falls in low-variance, strongly correlated directions. The proposal tests this claim directly.

Where the two literatures meet, and what is missing

The drift side is anchored by Right Now, Wrong Then: Non-Stationary Direct Preference Optimization under Preference Drift (NS-DPO). NS-DPO models preferences with a time-varying Bradley–Terry model, in which the probability of preferring one response over another depends on the difference of their rewards at time $t$. It weights each training pair by $\gamma^{T-t_i-1}$. This motivates our work in two ways. It shows that recency weighting rescues accuracy after preference flips. It also concedes that the right $\gamma$ depends on a total drift budget that nobody observes. Its discount is one scalar applied to all directions.

Two other methods regularize adaptation to drift but also share one rate across directions, or ignore time entirely. Continual Low-Rank Adapters for LLM-based Generative Recommender Systems (PESO) keeps one evolving low-rank adapter (a small trainable add-on to a frozen model). It anchors that adapter to its previous value with a softmax-KL penalty, which the authors show acts as an eigenvalue gate. The gate lets the adapter move along directions where the new data have high curvature and holds it still elsewhere. A Shared Low-Rank Adaptation Approach to Personalized RLHF splits the adapter into a projection shared across users and a small per-user part, but treats each user as static.

PESO is the closest existing mechanism, and the contrast matters. Its gate is keyed to where the new data are informative, not to where the preference has moved. Under strong feature correlation these can come apart. A preference can shift along a direction that the current window barely constrains, and PESO's gate would then freeze exactly the direction that needs to update.

The correlated-feature side supplies both the failure mode and the constraints on design. Work on ill-conditioned designs describes the core problem: many weight vectors fit the data equally well, estimates flip sign under small perturbations, and sparse selection becomes arbitrary. This is documented in Instability and Identifiability in High Dimensional Multicollinearity, Evaluating Variable Selection Techniques for Multivariate Linear Regression and Semi-Analytic Resampling in Lasso.

Graph-Based Regularization for Regression Problems with Alignment and Highly Correlated Designs stabilizes the problem by forcing correlated features to take similar weights. Its authors name the limitation themselves: this breaks when correlated features have different effects. Drift can create exactly that situation, for example when a user tires of one of two correlated attributes. We therefore avoid any penalty that assumes correlated features share weights.

Bayesian Factor-adjusted Sparse Regression separates dense common factors from sparse leftover structure. It shows that plain principal-component truncation loses predictive signal that lives outside the top components. That result is the reason we do not simply identify "stable directions" with "top principal components."

Core Claim

Write the reward as linear in a frozen representation, $r_t(x,y)=\theta_t^\top\phi(x,y)$. Here $\phi(x,y)\in\mathbb{R}^p$ is a fixed embedding of prompt $x$ and response $y$ from a pretrained language model, and $\theta_t$ is the preference vector at time $t$. We model drift as a random walk confined mostly to a low-rank subspace:

$$\theta_t=\theta_{t-1}+U\eta_t+\epsilon_t,\qquad \eta_t\sim\mathcal N(0,S),\ \epsilon_t\sim\mathcal N(0,\sigma^2 I).$$

In this model, $U\in\mathbb{R}^{p\times k}$ has orthonormal columns spanning the drift subspace, $S$ is a $k\times k$ drift covariance, and $\sigma^2\ll\|S\|$ allows small drift elsewhere. Under a Laplace-approximate Kalman filter, the uncertainty in $\theta$ is inflated between time steps by $\Sigma_{t|t-1}=\Sigma_{t-1}+USU^\top+\sigma^2 I$. In the filter, old evidence acts as a prior. Inflating uncertainty along $U$ therefore discounts old evidence only in that subspace.

Scalar exponential discounting is the special case $\Sigma_{t|t-1}=\Sigma_{t-1}/\gamma$. That update inflates uncertainty most in the directions where it is already largest, which are the low-variance, correlated directions. This is the formal version of the tension between the two literatures: uniform forgetting spends its variance budget in exactly the wrong place.

The prediction follows from a bias–variance split. Error in the stable complement of $U$ should scale with the full sample size rather than the discounted effective size $\sum_i\gamma^{T-t_i-1}$. Bias from drift is controlled only inside the $k$-dimensional subspace. The gain over scalar discounting should therefore grow with two quantities: the condition number of the feature Gram matrix, and how little the drift subspace overlaps with the top principal components.

Research Program

The method has three components.

First, we estimate the drift subspace online. At each window, we compute gradients of the Bradley–Terry loss on recent pairs at the current filtered estimate. We accumulate their outer products, whitened by the current posterior covariance, and take the top-$k$ eigenvectors as $U$. This choice has a clear rationale: directions where recent data systematically disagree with the running estimate are the directions of change.

Second, we set $S$ and $\sigma^2$ by maximizing the filter's one-step-ahead predictive likelihood. This replaces NS-DPO's unobservable drift budget with a quantity estimated from the data. We choose $k$ the same way.

Third, we carry the idea into full preference optimization. In a LoRA-based DPO run, the gradient contributions of old pairs are projected onto the complement of an adapter-space drift subspace, and are discounted only inside it. This stage is secondary. It is pursued only if the linear-head results hold.

The analysis component derives the error decomposition above for the linear-Gaussian case. We will state explicitly when directional discounting reduces to scalar discounting, namely when drift is isotropic or $k=p$. That case is where we expect no gain, and we will report it.

Baselines

The naive reference point is a static Bradley–Terry reward head, trained with ridge regularization on all pairs with equal weight and tuned on a held-out split. This is the analogue of standard DPO. NS-DPO reports that DPO's reward accuracy collapses to roughly 50–55% after mid-dataset preference flips on UltraFeedback-RM and TV-HH (Right Now, Wrong Then).

The strongest existing method is NS-DPO-style scalar discounting. We apply it to the same linear head, with ridge regularization and $\gamma$ tuned jointly. In its native form, a full-model DPO run on Llama-3.2-1b, NS-DPO reports reward accuracy above 60–70% after flips, gains of over 10% in reward accuracy under gradual drift, and a length-controlled win rate of 8.93 versus 7.29 for DPO on UltraFeedback-LM. We will reproduce its reward-accuracy setting on the same datasets and change points.

For recommendation, the strongest competitor is PESO. It reports average gains of 3.7–6.3% on Hit@5/10 and NDCG@5/10 over cumulative and unregularized LoRA on Amazon Reviews 2023 and Yelp, using a 60% pretraining block followed by four 10% chronological stages. We will use that exact split.

All methods receive the same frozen embeddings, the same windows, and the same hyperparameter search budget. That budget is 30 configurations per method, selected on the last validation window only. Beating a baseline means higher post-change reward accuracy, or Hit@10 and NDCG@10, averaged across change points and seeds, with non-overlapping 95% bootstrap intervals, and no loss on stationary control streams.

Three ablations isolate the key ingredient, which is estimating where drift happens. The first replaces $U$ with a random subspace of equal rank. The second replaces $U$ with the top-$k$ principal components of the features. The third uses an oracle $U$ available in synthetic and constructed-flip settings. The second ablation also serves as the factor-model alternative a reviewer would raise.

Why not just…?

Why not just add strong ridge or elastic-net regularization to scalar discounting? Ridge fixes conditioning by shrinking all poorly constrained directions toward zero. When drift moves weight into a low-variance direction, ridge biases exactly the coordinate that needs to grow. The lasso component selects arbitrarily among correlated features, with only about 30% overlap in selected features across runs according to Graph-Based Regularization. This alternative is a real competitor, which is why it is the tuned baseline rather than a strawman. The synthetic experiment below, which sweeps where drift falls, determines whether it suffices.

Why not just use PCA plus a per-component forgetting filter? This is the most serious threat, because it is directional forgetting in a fixed basis. It is optimal if drift aligns with feature variance, and Bayesian Factor-adjusted Sparse Regression already shows that predictive signal often sits outside the top components. If this ablation matches our method on real data, we will conclude that the drift subspace is not needed there, and we will report that result.

Why not just prompt a long-context model with the whole interaction history? In AcCoRD, frontier agents reach under 30% perfect outcomes, and scores on triggered preferences fall as low as 0.09. When Rubrics Backfire shows judge accuracy dropping by up to 27.9% under instruction-level drift. In-context tracking is therefore not a reliable substitute. We will still include an in-context reward-scoring baseline on the 2C NSGO opinion-shift data for completeness.

Proposed Experiments

The decisive experiment is synthetic. We use real embeddings from UltraFeedback pairs and plant a known $\theta_t$ whose drift is placed, in separate runs, in the top principal components, in the bottom components, or in random directions. We sweep the condition number of the Gram matrix by subsampling correlated feature clusters.

The claim predicts that directional discounting ties with the PCA-basis filter when drift lies in the top components. It should win by a margin that grows with the condition number when drift lies in the bottom components. It should tie with scalar discounting when drift is isotropic. If there is no gain in the bottom-component setting, the idea fails.

The real-data experiments use UltraFeedback-RM, TV-HH and 2C NSGO with NS-DPO's change points, and Amazon 2023 (Musical Instruments, Movies & TV, Books) plus Yelp with PESO's chronological split. For recommendation, per-user linear heads sit on frozen item embeddings.

We report the following metrics:

  • post-change reward accuracy, and recovery time in windows;
  • accuracy on stationary controls, where the method must match the static baseline, as NS-DPO does;
  • ranking metrics;
  • coefficient sign stability across bootstrap resamples;
  • the principal angle between the estimated $U$ and the oracle, and between $U$ and the top principal components.

A small principal angle between $U$ and the top principal components on real data would signal that drift and variance coincide, and that the simpler filter is enough. Only if the linear results hold do we run the projected-gradient LoRA-DPO variant on UltraFeedback-LM. There we compare its length-controlled win rate against NS-DPO's reported 8.93, under matched compute.