Curvature Outlives Preference: Two-Clock Second-Order Tracking of Drifting Human Feedback

When people's preferences change, a model trained on their feedback has to forget old labels quickly. The current fix, exemplified by Right Now, Wrong Then (NS-DPO, which down-weights each preference pair exponentially by its age), forgets everything about old data at one rate. This note starts from a simple algebraic fact about pairwise preference losses. The part of the loss curvature that second-order methods actually use does not depend on which response was labeled "preferred." It depends only on which pairs were shown and on the model's current predictions. If a user or population flips its preference, the labels become wrong, but the old data still correctly describes the geometry of the loss surface.

The proposal is to track drifting preferences with two clocks. The gradient is estimated from a short, heavily discounted memory, because gradients carry the labels and go stale. The curvature is estimated from a long memory that keeps even obsolete-labeled data, because curvature is label-free and stays valid. Taking a Newton-style step, which rescales the gradient by the inverse curvature, should let the model re-converge after a preference change using far fewer fresh labels than first-order discounting. Hessian smoothness (a bound on how fast curvature can change as parameters move) enters in two places. It decides how long a cached curvature estimate can be trusted as the parameters travel toward the new preferences. It also caps how large a post-change step can safely be.

Core Claim

Consider a Bradley–Terry or DPO-style loss on a pair with margin $z_\theta$, where the margin is the model's score difference between the two responses and the loss is $-\log\sigma(z_\theta)$ for the labeled winner. Its Hessian splits into two parts: a Gauss–Newton part and a residual part.

$$ \nabla^2_\theta \ell = \sigma(z_\theta)\big(1-\sigma(z_\theta)\big)\,\nabla_\theta z_\theta \nabla_\theta z_\theta^{\top} \;+\; \big(\sigma(z_\theta)-1\big)\nabla^2_\theta z_\theta . $$

Swapping the label sends $z_\theta \to -z_\theta$. The first (Gauss–Newton) term is unchanged, because $\sigma(1-\sigma)$ is symmetric and the outer product is sign-invariant. Only the residual term and the gradient change. Pure preference drift, where the same kinds of pairs appear but people now pick the other response, therefore leaves the Gauss–Newton curvature at a fixed parameter exactly invariant. Only a shift in which prompts and responses appear (covariate drift) can move it. Magnitude drift in rewards also leaves it invariant at fixed parameters, because $\sigma(z_\theta)$ is the model's own prediction, not the label.

The claim, which the experiments must confirm, is that decoupling memories exploits this asymmetry. Let $g_t$ be a gradient averaged with a short discount $\gamma_g$ as in NS-DPO. Let $\bar H_t$ be a Gauss–Newton average with a much longer discount $\gamma_H \approx 1$. The update is

$$ \theta_{t+1} = \theta_t - \big(\bar H_t + \rho_t I\big)^{-1} g_t, $$

where $\rho_t$ is an adaptive regularizer set from an online estimate of the Hessian Lipschitz constant. This update should recover from change points with markedly fewer post-change labels than NS-DPO or a single-clock second-order method.

The prior work plays different roles here. NS-DPO motivates the work: it shows drift breaks DPO and leaves the discount factor tied to an unknown drift budget. NASketch supplies the machinery: an online Newton method with running Hessian averaging and a sketched solver at $O(d^2)$ cost. Its unweighted cumulative Hessian average is exactly what our analysis says is safe for the label-free curvature part and unsafe for everything else. ARNCG supplies the adaptive regularizer and the online Lipschitz-estimation subroutine. It was built for static objectives, and we repurpose it to size steps after a change point.

Research Program

The first component is analysis. We will show that in the linear-in-parameters case (a reward head on frozen features), the two-clock update tracks a piecewise-constant drifting optimum. Its post-change error should depend on the number of fresh labels only through the gradient's variance. The curvature error, by contrast, shrinks with the total number of pairs ever seen. We expect this to yield a recovery bound whose fresh-sample dependence is better than first-order discounting by roughly a condition-number factor, since preconditioning removes ill-conditioning from the rate. We will state precisely what breaks under covariate drift.

For deep models we use the Gauss–Newton matrix rather than the full Hessian. The residual term is label-dependent. It is also the only place where ReLU-type non-smoothness enters, and prior work shows non-smoothness violates Hessian Lipschitz assumptions: Agarwal et al. and Yuan et al., the latter showing that even weak convexity fails empirically.

The second component is the Hessian-smoothness mechanism. Cached curvature was computed at old parameters. If the Gauss–Newton map has Lipschitz constant $L_2$, the staleness error of a cached term from time $s$ is at most $L_2\|\theta_t - \theta_s\|$. That error is near zero in stationary periods and large exactly after a change point, when parameters move fast. We will estimate $L_2$ online in the style of ARNCG's Lipschitz-estimation subroutine, by comparing curvature-vector products on a fixed probe set at successive iterates. The estimate serves two purposes. It triggers selective recomputation of old curvature at the current parameters. It also sets $\rho_t$ so that the step stays inside the region where the quadratic model is trustworthy. The Newton step immediately after a preference flip is large, and without this guard it overshoots.

The third component is engineering at scale. Full-parameter Newton on a 1B-parameter model is not feasible. We work in three parameter spaces of increasing size. The first is a linear reward head over frozen features. The second is PAL's low-dimensional user mixture weights, from PAL (a shared representation plus $K$ preference prototypes, adapted with about 20 samples per user). The third is LoRA adapters on Llama-3.2-1b, solved with conjugate gradient using Gauss–Newton-vector products. In all three, the long-memory curvature is stored as a sketch or recomputed on a replay buffer of old pairs whose labels are ignored.

Baselines

The naive reference point is standard DPO or standard reward-model training with AdamW on all data, with no discounting. On TV-HH with a mid-dataset flip, NS-DPO reports that DPO falls to "~50-55%" reward accuracy (Right Now, Wrong Then). Any method must clear this.

The strongest existing method is NS-DPO itself: exponential age discounting of the DPO loss, optimized first-order. It reports ">60-70%" reward accuracy on TV-HH under flips, more than a 10% reward-accuracy gain under gradual drift, and an AlpacaEval 2.0 length-controlled win rate of 8.93 versus 7.29 for DPO on UltraFeedback-LM. It is the strongest competitor because it is cheap, has a dynamic-regret guarantee, and costs nothing when there is no drift. For fairness, we will tune its discount factor on the same validation stream and with the same number of tuning trials we give our two discounts. We will also report an oracle-tuned NS-DPO that picks the best discount in hindsight.

Beating NS-DPO means one of two things at matched wall-clock compute. Either our method reaches NS-DPO's final post-change reward accuracy with at most half as many post-change labels, or it achieves higher accuracy at a fixed small post-change label budget, with no loss in the stationary segments.

For the recommendation setting, the strongest method is PESO from Continual Low-Rank Adapters. PESO evolves a single LoRA adapter with a softmax-KL proximal anchor that behaves like an eigenvalue-dependent gate, and it reports average gains of 3.7% to 6.3% on Hit@K and NDCG@K over competitors on Amazon and Yelp. We will run it on its own four-stage chronological splits with its code settings and the same adapter rank.

The ablation that isolates our key ingredient is a single-clock second-order method. It uses the identical optimizer, sketch, and regularizer, but the curvature uses the same short discount as the gradient. If this ablation matches the two-clock method, then the benefit comes from preconditioning alone and not from the label-invariance insight. A second ablation removes the Lipschitz-based regularizer and staleness refresh, to test whether the Hessian-smoothness guard matters after flips. A third uses the full Hessian in place of Gauss–Newton, which the theory predicts should hurt, since the residual term flips sign with the labels.

Why not just…?

Why not just tune NS-DPO's discount, or add change-point detection and reset? This is the most serious threat, especially for abrupt flips, where detecting the change and discarding old data might already recover quickly. The file offers no evidence that a detector-plus-reset fails. The honest answer is that a reset throws away curvature information that remains valid. Our main experiment varies the number of post-change labels from very few to many. A reset is competitive when fresh labels are plentiful. The claim fails if the reset matches us at the smallest label budgets. Resetting also hurts users who return with old preferences, which PESO documents as the "dormant user" failure of plain fine-tuning.

Why not just use Fisher-weighted anchoring toward old parameters, as in elastic weight consolidation? Such anchoring uses curvature to pull the model back toward its previous solution. Under a preference flip, that previous solution is the wrong one, so the anchor actively resists the needed change. Our method uses curvature only to shape the step, never to anchor. PESO's KL gate is a first-order cousin of the anchoring approach. We will include a Fisher-anchored variant alongside PESO on TV-HH flips, where our analysis predicts it underperforms even NS-DPO.

Why not just use Adam, whose second-moment statistics already look like curvature? Adam's per-coordinate squared-gradient estimate is diagonal. It also depends on the labels, because the squared gradient carries the squared residual. It therefore goes stale under drift in exactly the way gradients do and cannot pool obsolete data. All NS-DPO baselines already run AdamW, so the comparison against them tests this directly.

Why not just a single-clock second-order optimizer such as a discounted NASketch? This is the ablation above. It is the cleanest test of whether curvature memory should outlive gradient memory.

Proposed Experiments

The first check comes before any method comparison, because it can falsify the premise cheaply. On TV-HH, UltraFeedback-RM, and 2C NSGO, we fix the parameters and measure two changes across each change point. One is the overlap of the top Gauss–Newton eigenspaces before and after, using Gauss–Newton-vector products. The other is the change in gradient direction. The theory predicts near-invariant curvature and large gradient change for label-flip drift. If these benchmarks also shift which prompts appear, so that curvature moves substantially, the long curvature clock must shorten and the expected gain shrinks. We will report this directly. For 2C NSGO, no reported result was found on Lacuna, so we establish our own NS-DPO numbers there.

The main experiment trains reward heads, PAL mixture weights, and Llama-3.2-1b LoRA adapters on TV-HH and UltraFeedback-RM with abrupt and gradual drift. We plot reward accuracy against the number of post-change labels. We add UltraFeedback-LM for generation quality, measured by length-controlled win rate against NS-DPO's reported 8.93.

To test pluralistic drift, we inject drift into Pick-a-Filter and Reddit TL;DR by moving users between preference groups over time, using the PAL codebase RamyaLab/pluralistic-alignment. This measures few-shot re-personalization after a user switches groups. PAL reports that about 20 samples suffice for a new user, and we ask how many a drifting user needs.

The final experiment is Amazon and Yelp under PESO's protocol, including its dormant-user and new-user splits. This is the regime our theory flags as hardest, because user and item distributions shift and the curvature itself moves. Here the Lipschitz-triggered refresh must earn its keep. A result at parity with PESO would still be informative. A clear loss would bound where the two-clock idea applies.

We will report compute honestly, as curvature-vector products and wall-clock time per update, since the method's value depends on its extra cost staying small relative to what labels cost to collect.