When a chatbot gets something wrong, users rarely click "thumbs down." Only 1–3% of users give explicit ratings (DRIFT). Instead they complain, rephrase, or correct: "no, shorter," "I said Japan, not the US." Those corrections are a free training signal. But a correction can mean two different things. The model may have never learned what this user wants. Or it may have learned it correctly and the user has since changed their mind. Treating both cases the same way breaks personalization. If every correction is just another training example, old habits drown out new intent. If old data is simply forgotten at a fixed rate, stable preferences get erased along with stale ones.
We propose a way to tell these two cases apart, one conversational turn at a time, with no stored history. Each user gets a small personal adapter on top of a shared language model. When a correction arrives, the system checks whether the failed response came from the shared model or from what the personal adapter had learned about this user. If the adapter caused the failure, that is evidence the user's preference has shifted. The system then erases the specific part of the adapter responsible before learning the new preference. If the shared model caused it, the system simply learns. Forgetting is triggered by blame, not by the passage of time.
Core Claim
The claim is that one comparison separates "the model doesn't know yet" from "the user changed." The comparison asks how much the personal adapter shifted probability toward the response the user rejected, relative to the shared model. This quantity can be computed from a single correction with two forward passes. Using it to gate selective forgetting should recover faster after preference changes than time-based discounting. It should also preserve preferences that did not change, without anyone knowing in advance how fast each user drifts.
Two terms need defining. Direct preference optimization (DPO) is a method that trains a model directly on (preferred, rejected) response pairs, with no separate reward model. A low-rank adapter (LoRA) is a small set of trainable weights added to a frozen model. Let $\pi_0$ be the shared model and $\pi_\theta$ the shared model plus user adapter $\theta$. Let $y_l$ be the response the user corrected and $y_w$ a revised response generated by conditioning on the correction. The blame score is
$$ d_t = \big[\log \pi_\theta(y_l\mid x) - \log \pi_0(y_l\mid x)\big] - \big[\log \pi_\theta(y_w\mid x) - \log \pi_0(y_w\mid x)\big], $$
where $x$ is the conversation context. A large positive $d_t$ means personalization pushed the model toward the failure. Before taking the usual single DPO gradient step on $(x, y_w, y_l)$, we remove the adapter's component along the direction that produced this blame:
$$ \theta \leftarrow \theta - \alpha\,\sigma(\kappa d_t)\,(\theta^\top \hat g_t)\,\hat g_t, \qquad \hat g_t = \nabla_\theta d_t / \lVert \nabla_\theta d_t \rVert . $$
Here $\sigma$ is the logistic function, $\alpha \in [0,1]$ is the maximum erasure fraction, and $\kappa$ sets sensitivity. Only the offending direction is reset toward the shared model. The rest of what the adapter knows about the user is left intact.
Where This Sits
Three lines of prior work each supply one piece and leave the join open.
The first line learns from implicit feedback. DRIFT treats real failed turns as rejected responses and samples improved responses conditioned on the user's critique. It trains with multi-round offline DPO. Retrospective Learning from Interactions uses a frozen LLM to decode interaction logs into positive, neutral, or negative feedback with over 90% precision, then retrains between deployment waves. Both motivate our signal and supply our pair construction. Both are batch methods, and both assume the target preference is fixed across rounds.
The second line models drift. Non-Stationary DPO discounts older explicit pairs by $\gamma^{T-t_i-1}$, a weight that shrinks with age. It constrains our design in two ways. It shows that stationary DPO collapses to near-chance accuracy under a preference flip. Its own discount needs a known drift budget and fades all past data uniformly. Our gate replaces the time-based discount with a feedback-driven one.
The third line learns from streams. Online-LoRA detects distribution shifts from loss plateaus and resets adapters, but only on vision tasks with clean labels. With noisy conversational feedback, loss curves may never plateau. Amphibian scales updates by how well gradients agree across samples. Our blame gradient is a targeted relative of that idea. Instead of measuring interference among samples, it asks which component caused a failure. That question echoes failure-point isolation in voice assistants (FPI), applied to shared versus personal weights instead of speech recognition versus language understanding.
Two further papers matter. T-POP adapts online, but requires users to pick between two responses and never updates the language model itself. The early dictation system in Unsupervised learning from users' error correction learned online from edits, but filtered out "changes of mind" as noise. In our setting those changes of mind are exactly the signal to track.
Research Program
The system has three parts.
The first part is a causal feedback decoder. It is a frozen LLM that reads only an assistant turn and the next user turn, and labels the pair as friction, satisfied, or neutral. This is a causal restriction of the retrospective decoder in Retrospective Learning from Interactions: it never looks further into the future. We will measure its precision on the follow-up corrections in ExpectBench (Expectation Alignment).
The second part is the update. On each friction turn, we build a DRIFT-style pair by regenerating a response from the current policy conditioned on the user's correction. We then compute $d_t$, apply the gated erasure, and take one DPO step on the user's adapter. The turn is then discarded. Satisfied turns get a small update that pulls the adapter toward reinforcing the accepted response. This follows RESPECT's use of positive signals.
The third part is the analysis. We will check that $d_t$ separates true flips from first-time learning on simulated streams where ground truth is known. We will also characterize failure modes: decoder false positives, adversarial users, and users whose corrections are internally inconsistent.
The models are LLaMA-3.1-8B and Mistral-7B, the backbones used for ExpectBench. Each user has a rank-8 adapter.
Baselines
The naive reference is the shared model with no personalization. Next above it is plain online DPO: one gradient step per friction pair, no forgetting. This is what DRIFT's pair construction reduces to under a single-pass constraint.
The strongest drift-aware competitor is online NS-DPO. Each adapter update applies decay equivalent to the $\gamma$ discount, with $\gamma$ tuned by grid search on held-out users under the same budget we give $\alpha$ and $\kappa$. We also run an oracle version where $\gamma$ is tuned separately for each user's true drift rate. On its own offline benchmarks, NS-DPO keeps accuracy above 60–70% under mid-stream preference flips, where standard DPO falls to about 50–55%. It also reaches an AlpacaEval 2.0 length-controlled win rate of 8.93 versus 7.29 for DPO on UltraFeedback-LM (Non-Stationary DPO). No reported result found on Lacuna for NS-DPO, or any method, on a single-pass stream of implicit conversational feedback. Beating it means lower cumulative friction and faster recovery than tuned global $\gamma$, while retaining unchanged preferences at least as well. Matching the per-user oracle $\gamma$ without being told drift rates would count as strong success.
The non-single-pass reference is periodic DRIFT: logs are stored and offline DPO rounds are run at fixed intervals, with total gradient FLOPs matched to ours. DRIFT reports up to +6.23% WildBench task score at 7B (DRIFT). We treat it as an upper reference, because it violates the no-storage constraint.
Two ablations isolate the key ingredient. The first uses scalar erasure, shrinking the whole adapter by $\sigma(\kappa d_t)$ rather than only the blamed direction. It tests whether erasure needs to be selective. The second is learning-rate gating: the learning rate scales with $\sigma(\kappa d_t)$ and there is no erasure. It tests whether resetting to the shared model matters beyond simply taking bigger steps.
All methods see identical streams, use identical decoders and adapters, and get the same hyperparameter-search budget.
Why not just…?
Why not just keep a running memory of the user's corrections in the prompt? This is a real threat, especially within a single session. AcCoRD finds that prompting alone yields fewer than 30% perfect outcomes, but that result concerns eliciting preferences users have not yet formed. It does not concern remembering corrections they have already given. Over long, multi-session histories, a memory holds contradictory corrections side by side, and the model must resolve them in context without being told which is current. We run a retrieval-memory baseline of summarized past corrections. We report results by stream length and by number of preference reversals. If memory matches our method on long streams with many flips, parameter updates are unnecessary for this regime and we will say so. We also test the two combined.
Why not just use a well-tuned decay rate? That is online NS-DPO, and it is the key comparison. The argument against it is heterogeneity. Real users drift at different rates, and a single $\gamma$ trades slow recovery for some users against forgetting for others. Our simulated population deliberately mixes stable users with fast-flipping ones. If tuned global decay equals blame gating on this mixed population, the idea fails.
Why not just store the logs and retrain nightly? When storage is allowed, this is the right choice, and we report it as an upper reference. Single-pass learning matters in two situations: on-device or privacy-restricted deployments, and when a correction should affect the very next turn rather than tomorrow's model. We measure the within-day friction that nightly retraining cannot remove.
Why not just steer a frozen model with a small online reward model, as T-POP does? T-POP needs the user to choose between two full responses and doubles per-token latency, from 0.09s to 0.18s (T-POP). We run it with a simulated user who makes those pairwise choices, giving it strictly richer feedback than ours. It serves as a reference: if it wins, that shows the cost of learning from single-output complaints.
Proposed Experiments
No existing benchmark combines complaint-style feedback, drift, and streaming, so we build one from existing parts.
The main testbed is a population of simulated users. Each user's hidden preference over T-POP's style attributes (verbose, concise, creative, uplifting) or NS-DPO's 2C NSGO opinion targets changes at unknown, user-specific times. At a change, some attributes flip and others stay fixed. An LLM user simulator writes a natural correction whenever a response violates the user's current preference, and otherwise continues the conversation. Prompts come from UltraFeedback. A GPT-4o judge scores responses against the user's current hidden preference; ExpectBench reports this style of judge correlating with human annotators at r = 0.963.
We report four metrics:
- cumulative friction rate over the stream;
- recovery time, meaning the number of turns needed after a change to return to pre-change satisfaction;
- retention, meaning satisfaction on attributes that did not change;
- satisfaction on stationary users, where any forgetting is pure cost.
Three checks are decisive:
- $d_t$ must be significantly larger at true flip points than at first-time learning events, measured by area under the curve against simulator ground truth.
- Directional erasure must beat scalar erasure on retention of unchanged attributes.
- Blame gating must beat tuned global decay on the mixed-rate population without losing on stationary users.
Two secondary tests probe the edges of the method. We inject decoder errors at 10–30% to test robustness. This mirrors the tolerance T-POP reports for contradictory feedback up to 30% of the time. We also run cross-episode AcCoRD-Travel for repeated simulated users, to see whether learned adapters reduce the triggered-preference failures that prompting leaves unsolved.
References
- DRIFT: Learning from Abundant User Dissatisfaction in Real-World Preference Learning
- Retrospective Learning from Interactions
- Right Now, Wrong Then: Non-Stationary Direct Preference Optimization under Preference Drift
- Online-LoRA: Task-Free Online Continual Learning via Low Rank Adaptation
- Amphibian: A Meta-Learning Framework for Rehearsal-Free, Fast Online Continual Learning
- FPI: Failure Point Isolation in Large-scale Conversational Assistants
- T-POP: Test-Time Personalization with Online Preference Feedback
- Unsupervised learning from users' error correction in speech dictation
- Expectation Alignment of Language Models for Real-World User Expectations
- AcCoRD: Evaluating User-Agent Collaboration Under Realistic User Preference Dynamics