People's preferences change over time, but rarely all at once. A user who starts preferring shorter answers usually still cares about correctness. A community whose taste in humor shifts toward one style still dislikes the same clichés. Current methods for learning from drifting preference data ignore this. The main example is Non-Stationary Direct Preference Optimization, Right Now, Wrong Then, which down-weights every old comparison by the same factor $\gamma^{T-t_i-1}$. A stale comparison is discounted equally whether it concerns a preference that has since flipped or one that has held steady for years. The data is therefore thrown away too fast along stable directions and too slowly along the ones that moved.
We propose to model drift as sparse jumps. A preference vector over a fixed set of features stays constant except for a small number of coordinates that change at a small number of times. The same sparsity that makes this a better model of drift also makes it cheap to fit with second-order methods, meaning Newton-type solvers that use the curvature of the loss rather than only its slope. The unknowns are the jumps, and almost all of them are zero. The curvature matrix that matters is therefore restricted to the few nonzero jumps. This is exactly the "second-order sparsity" exploited by SSNAL for the Lasso. The payoff is an online reward tracker that keeps old evidence wherever nothing changed, detects which preference directions moved and when, and updates at interactive latency.
Core Claim
Let $\phi(x,y)\in\mathbb{R}^p$ be a fixed feature map, for example the final hidden layer of a frozen reward model. Let each comparison $i$, observed at time $t_i$, say that response $y_{w,i}$ was preferred to $y_{l,i}$ for prompt $x_i$, and write $d_i=\phi(x_i,y_{w,i})-\phi(x_i,y_{l,i})$. We fit time-indexed preference vectors $\theta_1,\dots,\theta_T$ by solving
$$ \min_{\theta_1,\dots,\theta_T}\; \sum_{i} \log\!\left(1+e^{-\theta_{t_i}^{\top} d_i}\right) \;+\; \lambda \sum_{t=2}^{T} \lVert \theta_t-\theta_{t-1}\rVert_1 . $$
The first term is the Bradley-Terry likelihood, the standard model that the probability of preferring $y_w$ is a logistic function of the reward difference. The second term penalizes jumps, and $\lambda$ controls how much evidence is needed to declare one.
Substitute $\theta_t=\theta_1+\sum_{s\le t}\delta_s$, where $\delta_s$ is the jump at time $s$. The problem then becomes an $\ell_1$-penalized logistic regression over $(\theta_1,\delta_2,\dots,\delta_T)$, with a design matrix made of structured cumulative sums.
Three consequences follow. Old comparisons keep their full weight along every coordinate that has not jumped since they were collected. A brief anomaly that reverts costs $2\lambda$ per coordinate, so it is absorbed unless the evidence is strong. There is no variation budget to guess in advance, which is the tuning problem NS-DPO's authors flag for $\gamma$. The computational claim is that the Newton systems for this problem scale with the number of active jumps rather than with $pT$. That makes exact, high-precision refits cheap enough to run every time a new batch of feedback arrives.
The sparse-jump penalty itself is standard. The fused-lasso idea is old statistics, and we do not claim it. What is new is using it as the drift model for preference learning, in place of scalar discounting. The second new element is the observation that its sparsity pattern is exactly what active-set second-order solvers need, when run forward along real time rather than along a regularization path.
Research Program
The solver has an outer and an inner loop. The outer loop is a proximal Newton method: it replaces the logistic loss by its local quadratic approximation. QUIC and BIG & QUIC use the same strategy for sparse inverse covariance. The inner problem is then a weighted Lasso, which we solve with the semismooth Newton augmented Lagrangian method of Li, Sun and Toh. Its Newton system, reduced with the Sherman-Morrison-Woodbury identity, costs time in the size of the active set. That paper is a method source, and its limitation is also ours: it proves superlinear convergence only for the static squared-loss case. Extending its guarantees to the time-structured logistic case is part of the work, not an assumption. The column-dropping preconditioner of PD-PMM is our fallback if the inner systems prove ill-conditioned.
The streaming component reuses the Hessian Screening Rule in a new direction. That paper does three things along a decreasing sequence of regularization values on fixed data:
- It predicts the next active set from a second-order Taylor expansion over the current one.
- It warm-starts the solver from that prediction, cutting solver passes from 10–30 to 1–2.
- It maintains the inverse active-set Hessian with sweep operators.
Its authors state that the method is tied to such paths. We apply the same three steps along wall-clock time. When batch $T+1$ arrives, a new block $\delta_{T+1}$ is appended. The Taylor expansion of the current solution, restricted to the active set, predicts which coordinates will jump. The solver warm-starts from the previous solution with $\delta_{T+1}=0$.
The screening paper also names the weak point: memory grows as $O(|A|^2)$ if the active set keeps growing, where $|A|$ is the number of active variables. We handle this by folding jumps older than a horizon into a new fixed baseline. We eliminate them from the Newton system with a Schur complement, the same marginalization used to drop old landmarks in visual-inertial estimation. The active set is then bounded by the number of recent jumps.
Finally, we turn the fitted trajectory into something an alignment pipeline can use. Each old comparison receives a weight equal to the probability that $\theta_T$ and $\theta_{t_i}$ agree on the sign of $\theta^\top d_i$. These weights replace $\gamma^{T-t_i-1}$ in the DPO loss, leaving NS-DPO's training loop otherwise unchanged. This lets us test whether per-direction staleness beats scalar staleness downstream as well as at the reward level.
Baselines
The naive reference is a stationary Bradley-Terry reward model fit on all data with equal weights. Standard DPO trained on all pairs is the policy-level counterpart. NS-DPO reports that at change points this drops to roughly 50–55% reward accuracy. Beating it is necessary but uninformative.
The strongest existing method is NS-DPO's exponential discounting. At the reward level we use a discounted Bradley-Terry fit on the same features $\phi$. At the policy level we use NS-DPO itself. NS-DPO reports reward accuracy above 60–70% where DPO falls to about 50–55% at change points on TV-HH and 2C NSGO. It reports an AlpacaEval 2.0 length-controlled win rate of 8.93 against 7.29 for DPO on UltraFeedback-LM. It is the direct competitor on exactly our question, and it costs nothing extra.
For fairness:
- $\gamma$ receives the same validation-tuning budget as our $\lambda$, with the same number of grid points on the same held-out future slice.
- Both methods use identical features, backbones (Llama-3.2-1b and Llama-2-7b, as in NS-DPO), and data order.
Beating it means higher current-time reward accuracy and higher accuracy on held-out old pairs whose relevant directions did not drift, both over at least three seeds. At the policy level it means a win rate above NS-DPO's reported 8.93 under our reproduction of their setup. We will report our reproduced NS-DPO number alongside theirs.
Two ablations isolate the key ingredients. The first swaps the $\ell_1$ jump penalty for a squared one. That is a per-coordinate Gaussian random walk, fit by an extended Kalman filter, which is itself a cheap closed-form second-order tracker. This isolates sparsity in the drift model. The second fits our exact objective with a first-order solver: proximal gradient and ADMM at matched wall-clock budgets. This isolates the value of the second-order solver from the value of the model.
For the personalized setting on Reddit TL;DR we add P-ShareLoRA as a static per-user reference. It shares one projection matrix across users and learns a separate one per user, reporting 74.65% accuracy with Llama-3 8B. Here we test whether adding sparse per-user jumps on top of its user features helps annotators whose labels drift.
Why not just…?
Why not just tune NS-DPO's discount factor, or learn one discount per feature? This is the real threat. If drift in actual preference data is dense, meaning many directions move a little all the time, discounting is the right model and ours adds nothing. The per-feature-discount variant is close to our squared-penalty Kalman ablation, so it is covered. The experiment that settles this is synthetic. We plant drift with controlled density, from 1% to 100% of coordinates moving, on real $\phi$ features. We expect our method to win clearly when drift is sparse and to tie the Kalman baseline when it is dense. If it does not win in the sparse regime, the core claim is false. On real data, the number and stability of recovered jumps tells us which regime we are in.
Why not just solve the sparse-jump objective with Adam or proximal gradient and skip the Newton machinery? For offline use this may be enough. In that case the contribution is the model, and we will say so. The second-order solver matters when a refit must finish between conversational turns. It also matters because the output is the support itself, the list of which directions changed and when. Inexact first-order solutions blur that support. The evidence is indirect: SSNAL was at least 330 times faster than ADMM on ill-conditioned housing7, and coordinate descent stalled on madelon where subspace Newton did not (SSCN). Our matched-latency comparison, measuring change-point F1 and accuracy at 50 ms, 500 ms and 5 s per update, tests whether that transfers.
Why not just detect change points with a standard test such as CUSUM and refit on the data after the last change? Refitting after a global change point discards all old data, which is precisely the stable-direction evidence we want to keep. A per-coordinate change-point test followed by refitting is closer. However, it makes hard decisions that do not interact through the shared likelihood, and correlated features split one jump across several coordinates. The Hessian screening paper shows first-order rules degrade at feature correlation 0.8. We include the per-coordinate version as a baseline, so the experiments settle it directly.
Why not just put the history in an LLM's context? AcCoRD finds frontier models reach a perfect outcome less than 30% of the time under dynamic preferences. Its authors conclude that training-based approaches are needed. Our tracker targets the reward-learning side rather than dialogue, so we do not claim to address AcCoRD's triggered-preference failures.
Proposed Experiments
The first experiment is the decisive one: the synthetic drift-density sweep described above. It uses UltraFeedback-RM prompts and responses with planted piecewise-constant preference vectors. We measure current-time accuracy, change-point precision and recall, and stable-direction retention.
The second experiment reproduces NS-DPO's TV-HH, 2C NSGO and UltraFeedback-RM change-point and gradual-drift settings with their backbones. We report reward accuracy against their numbers and against our own reruns. We then run downstream weighted DPO on UltraFeedback-LM and evaluate with AlpacaEval 2.0 length-controlled win rate.
The third experiment uses naturally occurring drift. The New Yorker caption ranking data covers 365 weekly contests over eight years, drawn from Humor in AI. We train on contests up to week $T$ and predict caption rankings in week $T+1$, rolling forward through all eight years. There is no reported result on Lacuna for temporal drift modeling on this dataset, so all baselines will be run by us. It is also where we will inspect whether recovered jumps are few and interpretable.
The fourth experiment is the systems check. We measure per-update wall-clock time, active-set size over time with and without Schur-complement folding, and solver passes per update, and compare them to the 1–2 passes that Hessian screening achieves on static paths.
The project fails cleanly in either of two cases: sparse drift gives no advantage in the controlled sweep, or real datasets yield dense, unstable jump sets. Either result would itself be a useful finding about how human preferences actually move.
References
- Right Now, Wrong Then: Non-Stationary Direct Preference Optimization under Preference Drift
- A Highly Efficient Semismooth Newton Augmented Lagrangian Method for Solving Lasso Problems
- BIG & QUIC: Sparse Inverse Covariance Estimation for a Million Variables
- An Efficient Active-Set method With Applications to Sparse Approximations and Risk Minimization
- The Hessian Screening Rule
- A Shared Low-Rank Adaptation Approach to Personalized RLHF
- Cubic regularized subspace Newton for non-convex optimization
- AcCoRD: Evaluating User-Agent Collaboration Under Realistic User Preference Dynamics
- Humor in AI: Massive Scale Crowd-Sourced Preferences and Benchmarks for Cartoon Captioning