People change what they want, and systems that learn from their feedback must decide how fast to forget. They also have to cope with the fact that human feedback is noisy even when preferences are perfectly stable. These two situations look alike to a learner, because in both its predictions start missing. They call for opposite responses. Drift calls for forgetting faster. Noise calls for averaging over a longer history. Current methods cannot tell the two apart. A fixed recency discount treats every old data point as equally stale. A system that asks for help whenever its uncertainty is high asks just as often from a consistently noisy user as from one whose goals have just changed.
We propose a statistic that separates the two, drawn from a classical tool in reinforcement learning (RL). Recursive variance estimation learns the spread of future rewards with a Bellman-style update, the same one-step recursion that ordinary value learning uses for the mean. When the world is stationary and the variance estimate has settled, the variance learner's own one-step prediction error averages to zero. Persistent noise raises the estimated variance but leaves that error centered. Drift pushes the error systematically positive. We use this "variance surprise" to control how fast a recurrent user model overwrites its memory and how heavily old feedback is weighted. We then test whether this explicit, calibrated signal beats both fixed decay and a recurrent network left to learn its own forgetting.
Core Claim
The direct variance method of Sherstan et al. and its λ-return extension learns return variance with a cascade of two learners. A value learner $V$ produces the temporal-difference error $\delta_t = r_t + \gamma V(s_{t+1}) - V(s_t)$, where $r_t$ is the reward, $s_t$ the state and $\gamma$ the discount. A variance learner $\sigma^2$ then treats $\delta_t^2$ as its reward, discounted by $\gamma^2$. We look at the variance learner's own error:
$$\epsilon_t = \delta_t^2 + \gamma^2 \sigma^2(s_{t+1}) - \sigma^2(s_t).$$
When rewards are stationary and both learners have settled, the variance Bellman equation implies $\mathbb{E}[\epsilon_t \mid s_t] = 0$, however large the noise is. When the user's reward function moves, $V$ becomes biased. This adds a squared-bias term to $\delta_t^2$ that $\sigma^2$, learned slowly, has not absorbed, so $\epsilon_t$ has positive mean.
A running, scale-normalized average of $\epsilon_t$, which we call $z_t$, is therefore a drift detector with a built-in noise model. We feed it into forgetting through a single monotone map. The update gate of a recurrent belief network (a GRU, a recurrent cell whose update gate sets how much of its memory is overwritten at each step) receives an additive bias $b_t = a\,(z_t - c)$. The same signal sets a per-sample replay weight that replaces the fixed decay $\gamma^{T-t_i-1}$ of NS-DPO. Here $a$ and $c$ are two scalars shared across users.
When the reward breaks into attributes, as in AcCoRD's per-preference satisfaction or the meeting assistant's attribute filters, we keep one $(V, \sigma^2, z)$ triple per attribute. Stable core preferences then keep long memory while volatile ones are forgotten quickly. This addresses the uniform-penalty limitation that Right Now, Wrong Then acknowledges.
The claim is specific and falsifiable. On a factorial of drift rate × feedback noise, a fixed discount is optimal for only one cell, and a variance-magnitude trigger forgets fastest exactly where it should forget slowest, under high stationary noise. The variance-surprise controller should track the best fixed discount in every cell without being told the drift budget. It should keep that advantage at drift rates and noise levels not seen during training.
Relation to Prior Work
Two papers supply the method. Tamar et al. derived the linear second-moment Bellman equation and TD/LSTD variance estimators, and showed in Pinball that they beat Monte Carlo rollouts for a fixed sample budget. Sherstan et al. showed that the direct form avoids the negative variances and step-size fragility of the indirect $M - J^2$ form. Both evaluate a fixed policy in a stationary environment. We use their recursion as a test for whether stationarity holds.
Sherstan et al. also identify a weakness that cuts against us. The direct learner treats $\delta_t^2$ as ground truth, so a biased value function corrupts it. That corruption is precisely our signal. It also produces false alarms early in training, while $V$ is still learning, so the warm-up rule and the step-size ratio between learners become first-class design choices rather than footnotes.
Three papers motivate the problem. HULA asks a human for help when $M - Q^2$ exceeds a threshold. It found 80% agreement with Monte Carlo on the most uncertain states and solved partially observable grids with 4–8 expert calls, but it assumes a fixed expert. Its trigger responds to variance level, not variance surprise, so it cannot separate a noisy user from a changing one. Liu et al. document "concept evolution": users' preferences change in response to a meeting assistant's suggestions, the feedforward Q-network fails to converge, and median satisfaction falls to 4–5/7 against 7/7 for hand-written rules. That is the regime we target. AcCoRD shows frontier LLM agents scoring as low as 0.09 on preferences that only appear after users see options. It also shows that prompted uncertainty (UncReAct) does not help, and the authors point toward trained, stateful methods.
Two papers constrain the design. NS-DPO is the strongest existing drift method. Its fixed discount needs the drift budget in advance and applies uniformly across attributes. Preference variance shows that per-prompt variance bounds the DPO gradient and makes a useful data filter. It is computed in a static batch and says nothing about time, so we do not treat high variance as informative by itself. On the recurrent side, the forget-gate study shows the forget gate largely sets an LSTM's memory length. NeuralHydrology shows LSTMs widen and narrow their effective memory with the seasons. Together these say a learned gate can adapt its horizon to patterns it has seen. Our bet is narrower: it cannot calibrate against noise levels it has not seen.
Research Program
The work has three stages. In the first, we build the estimator and characterize it. In tabular and linear chain and random-walk MDPs of the kind Sherstan et al. used, we inject reward shifts and reward noise independently. We then measure detection delay and false-alarm rate for $z_t$ as functions of the step-size ratio between $V$ and $\sigma^2$. We also derive the stationary scale of $\epsilon_t$, which requires the fourth moment of $\delta_t$ and which we estimate online, so that $z_t$ has a usable threshold. The output of this stage is a recommended warm-up rule and step-size ratio, plus a clear statement of when the detector is blind. One known blind case is drift slow enough that $\sigma^2$ absorbs it.
In the second stage, we couple the detector to a recurrent agent. A GRU belief network sees the interaction history and feeds an actor-critic agent. The value and variance heads read the GRU state, and $z_t$ biases the update gate and sets replay weights. Each reward attribute gets its own triple. The two scalars $(a, c)$ are tuned once, on a single drift/noise condition, and then frozen.
In the third stage, we carry the idea to preference datasets. With one-step labels ($\gamma = 0$) the recursion collapses to a calibration check. For a Bradley-Terry model, which turns a reward difference into a choice probability $p$, stationary noise implies $\mathbb{E}[(y-p)^2] = p(1-p)$ for a binary label $y$. Squared error in excess of this marks drift. We use the resulting per-sample weight in place of NS-DPO's $\gamma^{T-t_i-1}$.
Baselines
The naive reference is a stationary learner: the same GRU agent with no added gate bias and uniform replay, and plain DPO on the preference datasets. It sets the floor that drift-aware methods must clear. NS-DPO reports this floor at about 50–55% reward accuracy after a mid-dataset preference flip.
The strongest existing method is fixed exponential recency weighting as in NS-DPO. In the interactive settings this becomes a fixed decay on replay weights and a constant gate bias. To make it as strong as possible, we give it an oracle: its discount is grid-searched separately for every drift/noise cell on held-out data, using the same tuning budget in total compute as we spend on $(a, c)$. On UltraFeedback-LM, NS-DPO reports a length-controlled AlpacaEval 2.0 win rate of 8.93 against DPO's 7.29. Under a sudden flip it keeps reward accuracy above 60–70%. Beating it means matching its per-cell-tuned performance within noise using a single frozen setting, and exceeding it on cells outside its tuning grid and on mixed-volatility attributes.
The variance-level baseline follows HULA's logic. Forgetting and clarification queries are triggered by $\sigma^2$ exceeding a threshold, using the direct estimator for a fair comparison. The prediction is that it over-forgets under high stationary noise.
The key ablation is a learned-gate GRU. It receives $\delta_t$, $\delta_t^2$ and $\epsilon_t$ as extra inputs and is trained on the same drift distribution, but has no hard-wired bias. This isolates whether the explicit coupling matters or only the information it carries. A second ablation replaces $z_t$ with an exponential average of raw $\delta_t^2$, with no variance learner subtracted. This tests whether the Bellman decomposition is doing the work. All agents share architecture, data, environment steps and seeds. We will use at least ten seeds and report confidence intervals.
Why not just…?
Why not just tune the decay factor on validation data? If drift velocity were known and constant this would be hard to beat, which is why the oracle-tuned NS-DPO is our main baseline. NS-DPO's own authors note that the right discount depends on an unknown drift budget and is uniform across attributes. Our factorial reports the per-cell-tuned decay alongside the single best global decay. If the frozen variance-surprise controller is no better than the global decay, the idea fails.
Why not just feed the TD error into an LSTM and let the gates learn? This is the real threat. Forget-gate and hydrology results show gated networks do learn horizon-adaptive memory. A learned gate may well equal ours when train and test drift distributions match. The test that settles it is the out-of-distribution split: train at low noise and moderate drift, then evaluate at high noise and fast or slow drift. If the learned-gate ablation still matches, the honest conclusion is that $\epsilon_t$ is a valuable input feature rather than a required control law, and we would report it that way.
Why not just ask the user whenever uncertainty is high, as HULA does? Variance level conflates noise with change. A consistently noisy user would be questioned constantly, which is the over-asking failure AcCoRD reports for weaker models. We measure queries per resolved preference and query precision, meaning the fraction of queries issued within a window after a true drift event, for both triggers.
Why not use Monte Carlo variance or a generic change detector on rewards? Tamar et al. show Monte Carlo variance is far less sample-efficient than recursive estimation, and full rollouts are not available with real users. A generic detector on raw rewards has no state-conditional noise model. The raw-$\delta_t^2$ ablation stands in for this family.
Proposed Experiments
The decisive experiment is the drift × noise factorial: four drift rates, including none and abrupt flips, crossed with four feedback-noise levels. It runs in a simulated meeting-assistant environment rebuilt from Liu et al.'s description, with 17 state features, 3–8 attendees with 8 attributes, and binary feedback, and with simulated users whose attribute weights drift. The metrics are cumulative regret against the drifting optimum, regret in the first 50 steps after a drift event, and false forgetting under pure noise. The success criteria are three. Our controller should fall within the oracle-tuned decay's confidence interval in at least 14 of 16 cells. It should beat the variance-level trigger in the high-noise rows. It should beat the learned-gate ablation on held-out cells.
The second experiment replicates NS-DPO's TV-HH, UltraFeedback-RM/LM and 2C NSGO splits, using the one-step calibration weight. We report reward accuracy around the change point and AlpacaEval 2.0 length-controlled win rate against the reported 8.93. We add a mixed-volatility variant of TV-HH in which helpfulness preferences flip while harmlessness stays fixed, to test per-attribute forgetting.
The third experiment is a stretch on AcCoRD-Shop. A small trained agent with the GRU belief and per-preference triples decides when to clarify, and we measure triggered-preference scores, perfect-outcome rate and turns used. The reference points are that frontier models score as low as 0.09 on triggered preferences, Claude-4.5 reaches a perfect outcome less than 30% of the time, and the best WebShop imitation-plus-RL agent reaches 29.1% success on the underlying store. We expect at most a modest gain here, and a null result would not sink the core claim. The core claim stands or falls on the factorial.
References
- Comparing Direct and Indirect Temporal-Difference Methods for Estimating the Variance of the Return
- Directly Estimating the Variance of the λ-Return Using Temporal-Difference Methods
- Right Now, Wrong Then: Non-Stationary Direct Preference Optimization under Preference Drift
- Temporal Difference Methods for the Variance of the Reward To Go
- Decision Making for Human-in-the-loop Robotic Agents via Uncertainty-Aware Reinforcement Learning
- Evaluating Rule-based Programming and ReinforcementLearning for Personalising an Intelligent System
- AcCoRD: Evaluating User-Agent Collaboration Under Realistic User Preference Dynamics
- On the Role of Preference Variance in Preference Optimization
- The unreasonable effectiveness of the forget gate
- NeuralHydrology - Interpreting LSTMs in Hydrology