When a model learns continually from people whose preferences change, it keeps a small replay buffer, a store of past examples that it re-trains on so it does not forget. That buffer can turn harmful. Examples labeled under yesterday's preferences keep pulling the model back toward yesterday. The usual fixes are blunt. They down-weight everything old, or they flush a whole class once a global statistic shifts. Both throw away old examples that are still correct.
This note proposes a more precise fix, made possible by one design choice. If inputs are encoded by a frozen pre-trained model, an encoder whose weights never change, then a stored example's position in feature space never moves. Only its label can go out of date. So each stored example can be checked against its own neighbors. If recent examples near it in the frozen space consistently disagree with its stored label, it is stale and should be evicted. If no recent example has landed near it, nothing has contradicted it, and it should be kept no matter how old it is.
The same fixed geometry fixes a known problem in embedding visualization. A 2D map computed once stays valid for the whole stream. Drift then shows up as labels changing color on a stable map, instead of a map that rearranges itself every time it is recomputed.
Core Claim
Preference drift in practice is usually local. A user tires of one genre but not of music. An annotator pool tightens safety norms on one topic but not on coding help. Under local drift, the right unit for deciding what to forget is a neighborhood, not a timestamp and not a class.
We claim that a frozen encoder makes neighborhood-level staleness cheap and statistically testable. A buffer that evicts only examples whose neighborhoods are contradicted by recent evidence should hold up better than two alternatives. It should beat exponential time decay on accuracy for current preferences, because it keeps persistent "anchor" preferences that decay discards. It should beat global drift tests, because it reacts to drift confined to a small region that those tests average away. The advantage should shrink to zero as drift becomes global, since there are then no unchanged regions to protect. That dependence is the decisive prediction.
Concretely, let $z_i$ be the frozen feature vector of buffered example $i$ and $y_i \in \{-1,+1\}$ its stored label. For preference pairs, $z_i$ is the difference of the two responses' features, and $y_i$ says which one was preferred. Let $N_k(i)$ be its $k$ nearest neighbors among the stream examples from the most recent window. The contradiction score is
$$c_i = \frac{\sum_{j \in N_k(i)} s_{ij}\, \mathbb{1}[y_j \neq y_i]}{\sum_{j \in N_k(i)} s_{ij}},$$
where $s_{ij}$ is the cosine similarity between $z_i$ and $z_j$, restricted to neighbors above a similarity floor.
Example $i$ is evicted when $c_i$ exceeds the disagreement rate expected from label noise alone, judged by a one-sided binomial test. That expected rate is estimated from neighborhoods during a calibration period known to be stationary. If example $i$ has fewer than a minimum number of recent neighbors, it is marked "unchallenged" and protected from drift-based eviction. When the buffer is full and nothing is contradicted, space is freed by dropping the most redundant example in the densest region, which keeps the buffer's coverage of the space.
The lightweight head on top, a linear Bradley-Terry reward model or a linear classifier, is retrained on the buffer plus the recent window.
Where This Sits
The proposal draws on three lines of work, each playing a different role.
The drift-aware replay papers motivate the problem and supply the closest competitors. Holistic Continual Learning under Concept Drift with Adaptive Memory Realignment names the "Replay Paradox": under drift, larger buffers hurt because they replay obsolete data. Its fix runs a Kolmogorov-Smirnov test on prediction uncertainty for each class and flushes that class's memory. This is per-class and global within a class, and the file notes it is insensitive to slow drift. Class-Incremental Experience Replay for Continual Learning Under Concept Drift is the nearest relative. Its Reactive Subspace Buffer groups stored examples into clusters around centers and uses drops in cluster purity to trigger relabeling or pruning. It differs from our proposal in three ways. It clusters in a backbone that is being trained, so cluster membership moves for reasons unrelated to drift. It judges purity inside the buffer rather than against fresh stream evidence. It was only evaluated on image streams where whole classes have their labels flipped. Unlearning-based sliding window for continual learning under concept drift removes the oldest chunk by approximate unlearning. That is age-based by construction, and it weakens on semantic drift.
The preference-drift work constrains what counts as a fair comparison. Right Now, Wrong Then: Non-Stationary Direct Preference Optimization under Preference Drift discounts each preference pair by $\gamma^{T-t_i-1}$, where $T$ is the current time and $t_i$ the time pair $i$ was collected. Its own stated limitation is that age cannot separate lasting anchor preferences from obsolete ones. That limitation is exactly the gap a neighborhood test targets. It also supplies benchmarks with known change points. MEGG selects replay examples for recommenders by how strongly their gradients align or conflict with the model's overall update direction. It has no notion of staleness, so extreme but obsolete examples can survive. A Shared Low-Rank Adaptation Approach to Personalized RLHF handles differences between users but treats each user's preferences as fixed over time.
The frozen-feature literature supplies the enabling assumption. Frozen encoders with shallow heads match or beat fine-tuning in low-data and shift-heavy settings: up to 21x faster domain adaptation in Zhang et al., and linear SVMs on frozen features in Kieffer et al.. The file also lists the cost. When concepts drift, frozen features cannot adapt, so all of the adaptation falls on the head and on the data the head sees. We turn that limitation into the mechanism. If the head carries the whole adaptation burden, then deciding which data the head sees is the central problem, and stable features make that decision tractable.
On the visualization side, Semantic Concept Spaces projects pre-trained word embeddings with t-SNE so that people can refine topics. It assumes a static corpus. WildVis shows that UMAP projections scale to millions of items. The file identifies recomputing projections on drifting embeddings as an open problem, because the map rearranges each time and users lose their bearings. Frozen features remove that problem. We use a single fixed projection as an evaluation and audit tool, not as the decision mechanism. We also test directly whether making decisions on the 2D map instead of the full feature space costs accuracy. Learning Embeddings into Entropic Wasserstein Spaces avoids projection by learning embeddings natively in 2D or 3D. That requires training the embedding, which conflicts with the frozen constraint, so we leave it aside.
Research Program
The system has four parts. The first is a frozen encoder: an open LLM's pooled hidden states for preference pairs, and a pre-trained vision backbone for the image streams. The second is a linear head retrained after every chunk of the stream. The third is the buffer with the contradiction-score eviction rule described above, using approximate nearest-neighbor search over the recent window. The fourth is a "drift atlas": a UMAP projection fit once on a reference sample, onto which each chunk's contradiction scores and evictions are drawn.
The analysis part of the program asks three questions. First, how does the advantage over time decay depend on how local the drift is? Second, how does the binomial test behave under gradual drift, where per-window evidence is weak? Here we accumulate evidence across windows with a simple running sum and measure detection delay. Third, does the method's advantage really come from fixed geometry? We answer this by running the identical rule on a backbone that is fine-tuned during the stream.
Baselines
The naive reference points are two. The first is reservoir sampling, a uniform random buffer with the same capacity, which the RSB paper shows collapsing toward 50–60% accuracy under label flips. The second is a pure sliding window that trains the head on recent data only. Both use the same frozen features and the same head.
The strongest competitor on accuracy for current preferences is exponential time decay in the style of NS-DPO, applied to the frozen-feature head. It is the method that explicitly targets preference drift. For context, the NS-DPO paper reports reward accuracy "above 60-70%" versus standard DPO's "~50-55%" after a preference flip on UltraFeedback-RM. On TV-HH it reports a margin of "over 10% in reward accuracy" over baselines under gradual drift. Those numbers come from fine-tuned policies, not frozen-feature heads, so we re-run the time-decay idea ourselves rather than compare against the reported figures.
Two more competitors are the per-class KS flush from AMR and RSB's cluster-purity rule. For the preference data, AMR's tests run per cluster, because preference data has no classes. RSB is applied on the same frozen features. Running RSB on frozen features is the hardest version of that competitor, because it gives RSB the same stable geometry our method relies on. Where the head is cheap enough, we also include full relearning on all data seen so far.
For fairness, every method gets the same encoder, head, buffer capacity, and recent-window size. Each gets an equal hyperparameter grid of twelve settings. That grid covers $\gamma$ for time decay; $k$ and the significance level for ours; the KS significance level for AMR; and cluster count and purity threshold for RSB. Hyperparameters are tuned on a held-out drift stream with a different seed and change-point schedule. Results are averaged over five seeds.
Three ablations isolate the key ingredients. The first replaces neighborhood evidence with the head's own disagreement with the stored label, which removes locality. The second computes neighborhoods in the 2D projection instead of the full space. The third fine-tunes the backbone during the stream, which removes geometric stability. Beating the time-decay baseline means higher current-preference accuracy, averaged over the stream, by more than two seed standard deviations on streams where at most a quarter of the space drifts. Within the same seed variation, we must also match it when drift is global.
Why not just…?
Why not just tune the decay rate well? Any single $\gamma$ trades off two errors. A short memory forgets unchanged regions. A long memory keeps stale labels in changed regions. When drift touches a small fraction of the space, no $\gamma$ fixes both, which is the limitation the NS-DPO paper concedes. The locality sweep tests this directly. If tuned decay matches our method even at 5% drift, the claim fails.
Why not just use RSB's cluster purity on frozen features? This is the most serious threat, because it gives RSB the same stable geometry. The remaining differences are two. Our evidence comes from fresh stream labels rather than buffer-internal purity: a buffer whose stale examples all agree with one another stays "pure" while being wrong. Our neighborhoods are also per example rather than tied to fixed cluster boundaries. If frozen-feature RSB matches our method, we will report that fixed geometry, not the specific rule, carries the benefit. That would itself be a useful and new finding, since RSB was only tested on trained backbones.
Why not just retrain the head on a recent window, since frozen features make retraining cheap? Cheap retraining removes any compute argument, so we make none. The question is which data to train on. A recent window throws away every unchanged preference that was not recently observed. Retention on unchanged regions is the metric that exposes this, and the sliding-window baseline measures it.
Why not just fine-tune the backbone? Fine-tuning lets features adapt to semantic drift that frozen features cannot express. That is a real weakness of our setting under the domain-mismatch limitation the file records. The moving-backbone ablation reports both the accuracy fine-tuning gains and how much the eviction rule degrades once neighborhoods start moving.
Proposed Experiments and Decisive Checks
The main testbed builds on NS-DPO's non-stationary UltraFeedback-RM splits and TV-HH. On top of the original change points, we flip the preference labels only inside chosen regions of the frozen feature space. These regions are defined by clusters of prompt features, and they cover 5%, 25%, 50%, or 100% of the data. We vary both abrupt and gradual onset.
The second testbed is RSB's label-flip image streams (MNIST through ImageNet10) and CLEAR-10's decade of natural drift, using a frozen vision backbone. RSB reports holding "above 91%" under flips where standard replay falls to 50–60%. That is a reference point for the trained-backbone setting, not a target directly comparable to frozen-feature runs. CLEAR-10 has no reported number on Lacuna. The third testbed is MovieLens 1M, with items encoded by a frozen text encoder over titles and genres and with time-ordered ratings. No reported result was found on Lacuna for MEGG's RMSE or AUC there, so we run MEGG's selection rule on the same features as an additional baseline.
The metrics are five. We measure accuracy on held-out data labeled under the current preferences. We measure accuracy restricted to regions that never drifted, which captures retention. We measure recovery in chunks after a change point, following UIL's measure. We measure the fraction of buffer slots holding stale labels, which is observable because the synthetic streams provide ground truth. Finally, we measure detection delay under gradual drift.
The study has three decisive checks. The first is the locality curve. Our advantage over tuned decay should be largest at 5% drift and vanish at 100%. A flat curve would mean locality is not the mechanism. The second is the moving-backbone ablation. If the advantage survives fine-tuning unchanged, then fixed geometry is not doing the work and the framing is wrong. The third is the atlas fidelity check. On synthetic streams, we measure how well regions flagged on the fixed 2D map overlap the true drifted regions, and how much accuracy the 2D-neighborhood ablation loses. This tells us whether the stable map is a trustworthy audit tool or a distorted picture to be used with caution.
References
- Holistic Continual Learning under Concept Drift with Adaptive Memory Realignment
- Class-Incremental Experience Replay for Continual Learning Under Concept Drift
- Unlearning-based sliding window for continual learning under concept drift
- Right Now, Wrong Then: Non-Stationary Direct Preference Optimization under Preference Drift
- MEGG: replay via maximally extreme GGscore in incremental learning for neural recommendation models
- A Shared Low-Rank Adaptation Approach to Personalized RLHF
- Efficient Pre-Trained Features and Recurrent Pseudo-Labeling in Unsupervised Domain Adaptation
- Convolutional neural networks for histopathology image classification: Training vs. Using pre-trained networks
- Semantic Concept Spaces: Guided Topic Model Refinement using Word-Embedding Projections
- WildVis: Open Source Visualizer for Million-Scale Chat Logs in the Wild
- Learning Embeddings into Entropic Wasserstein Spaces