Recency Hijacking: Adversarial Poisoning of Time-Discounted Preference Learning

When an AI system learns from human preference feedback over months or years, the humans' preferences change — society shifts, policies get updated, individual users grow and learn. A recent line of work responds by forgetting on purpose: time-discounted preference optimization weights recent feedback more heavily than old feedback, so the model tracks the present rather than the past. This proposal argues that this very mechanism opens a new attack surface. An adversary who wants to corrupt a model's safety alignment no longer needs to poison the whole history of feedback data; it only needs to inject a tiny batch of poisoned preference pairs into the most recent time window, because the discount factor exponentially amplifies exactly that window while aggressively fading out the clean historical record that would otherwise drown out the attack. We call this recency hijacking. The core of the proposal is to formalize the attack, prove that any purely timestamp-based weighting scheme is vulnerable to it, and build a defense that separates benign drift (which the system should embrace) from adversarial drift (which it must resist) — a distinction the current literature leaves entirely open.

The gap is sharp and, as far as the supplied literature shows, unexploited. Right Now, Wrong Then builds Non-Stationary DPO (NS-DPO), which weights each preference pair by an exponential factor $\gamma^{T-t_i-1}$ (with $\gamma \in (0,1]$, $T$ the current time and $t_i$ the pair's timestamp), and shows it rescues alignment when preferences shift — but it explicitly assumes drift is benign and exogenous, and the authors themselves flag that it "cannot distinguish between a high-quality old data point and a low-quality new one." Meanwhile, Preference Poisoning Attacks on Reward Model Learning shows that flipping just 0.3% of pairwise safety labels achieves a 100% attack success rate on LLaMA-7B safety alignment (on PKU-SafeRLHF-30k), and PoisonBench shows backdoors take hold at a log-linear rate from as little as 0.1% poisoned pairs while keeping a Stealthiness Score above 98% — invisible to standard evaluation. Nobody has yet put these two results together: under time discounting, that 0.1–0.3% becomes an effective poison fraction orders of magnitude larger, because the denominator of clean, heavily-weighted data shrinks to the recent window alone.

Positioning Against Closest Existing Work

Three papers sit nearest to this intersection, and none covers it. When Rubrics Backfire does connect misuse to preference drift, but its adversary edits an LLM judge's rubric prompt offline, once, statically — it does not touch a streaming, timestamped preference dataset and does not exploit any discount factor; it also reports that downstream DPO accuracy on harmlessness degraded by up to 27.9% while benchmark validation stayed clean, which motivates our choice of target-domain (not benchmark) evaluation. Right Now, Wrong Then supplies both our main victim method (NS-DPO) and our main evaluation dataset (TV-HH, a time-varying helpful–harmless split of Anthropic's HH-RLHF), but assumes no adversary. Theory of Continual Learning Against Data Poisoning Attacks proves the right kind of limits — no defense converges if a linear fraction of tasks is poisoned with unbounded noise (its Theorem 3.1), and a Task-to-Task verification scheme achieves 100% detection on CIFAR-10 — but its analysis is confined to supervised classification under linear/NTK approximations, with no notion of pairwise preference objectives or discounting. Our proposal ports its impossibility intuition into preference learning and shows discounting makes the picture strictly worse, then builds the defense its setting never needed.

Core Claim

The claim has three parts. First, attack: for any exponential discount factor $\gamma < 1$, the effective clean sample size against a recent-window poisoning attack is not the full history length $T$ but roughly $1/(1-\gamma)$ — so an adversary who would need to poison a fixed fraction of a stationary dataset to succeed needs only a poison budget proportional to a single time window, and poisoning a recent window dominates poisoning the distant past by a factor of $\gamma^{-\Delta t}$. Second, impossibility: no defense that assigns weights based on timestamps alone (any monotone recency weighting, of which NS-DPO is one instance) can be simultaneously responsive to genuine drift and robust to recency hijacking; this follows the spirit of Theorem 3.1 in Theory of Continual Learning Against Data Poisoning Attacks, transplanted to the discounted regime. Third, defense: robustness requires content-level information — the adversary's signal is not when data arrives but that its label flips are unexplained by any observable shift in the underlying conversation distribution — and a defense that combines window-level robust aggregation with a drift-consistency gate can achieve both goals.

Research Program

Attack construction. We formalize the adversary as a poisoning agent with a budget of $k$ label-flipped preference pairs per time step, choosing when to inject. On PKU-SafeRLHF-30k (used by Preference Poisoning Attacks on Reward Model Learning) and on PoisonBench-style backdoor triggers, we instantiate three variants: a naive stationary poisoning attack (spread evenly over history, the existing setting), an oracle recency attack (all poison in the final window), and a stealthy recency attack that mimics the TV-HH drift direction so the poisoned flips look like the natural drift toward helpfulness. We expect the attack-success-rate-versus-poison-budget curve under NS-DPO training to shift dramatically leftward relative to stationary training; the question the experiments answer is how far.

Theory. We prove the effective-sample-size and dominance results above in a Dynamic Bradley–Terry model (the time-varying preference model introduced by Right Now, Wrong Then), and a lower bound showing that for any timestamp-only weighting scheme there exists a poison budget within one window that moves the learned preference parameter by a constant amount while an undrifted, undiscounted learner requires a linear-in-$T$ budget. We also characterize when the stealthy recency attack becomes information-theoretically indistinguishable from genuine drift, which delimits what any defense can promise — the analogue of the indefensibility boundary in Theory of Continual Learning Against Data Poisoning Attacks.

Defense: drift-consistency-gated discounting. The defense replaces the pure timestamp weight with a product of recency and trust:

$$w_i \;=\; \gamma^{T-t_i-1}\,\sigma\!\big(\alpha\, c_i\big), \quad c_i = 1 - \frac{\left| \hat\Delta_{\text{label}}(i) - \hat\Delta_{\text{cov}}(i) \right|}{\hat B}$$

where $\hat\Delta_{\text{label}}(i)$ measures how much sample $i$'s preference direction departs from its window's consensus, $\hat\Delta_{\text{cov}}(i)$ measures how much the underlying prompt/response distribution shifted between adjacent windows (estimated from text embeddings, so a genuine societal or policy shift that moves both content and labels coherently earns $c_i \approx 1$), and $\hat B$ is an estimated variation budget — the quantity that Right Now, Wrong Then lists as unknown a priori, which we estimate online by tracking the covariate drift itself rather than assuming it. On top of the gate, each time window's gradient contribution is aggregated by a coordinate-wise median across sub-batches, a standard high-breakdown estimator that resists an attacker who controls a minority of sub-batches. The gate handles dispersed poison; the median handles concentrated poison. The critical honesty point: PoisonBench reports Stealthiness Scores above 98%, meaning poisoned pairs are individually indistinguishable from clean ones — so the gate deliberately operates at the window-aggregate level (does the flip pattern match the content drift?), not the per-pair level where the attacker wins.

Baselines

The naive reference point is doing nothing about drift: stationary DPO trained on the full pooled history. It is the strongest competitor on robustness alone, because with no discounting, a recent-window poison batch of size $k$ is diluted by all of history — this baseline tells us exactly how much of the vulnerability is created by the recency mechanism itself. It will be run with the same base model, data, and tuning budget as every other method. The strongest existing method on the combined question is NS-DPO from Right Now, Wrong Then, which is the best published drift-adaptive preference learner we are aware of; it reports maintaining reward accuracy of 60–70% under a sudden preference flip on TV-HH where standard DPO drops to roughly 50–55%, and an UltraFeedback-LM length-controlled win rate of 8.93 versus 7.29 for standard DPO. We will run NS-DPO across a full sweep of $\gamma$ under each attack variant, with identical data ordering, poison budgets, compute, and hyperparameter search budget as our defense. Beating NS-DPO means, concretely: at matched clean-drift reward accuracy on TV-HH (within 2 points of NS-DPO's), achieving a target-response selection rate for the attacker's chosen policy at least 5× lower under the stealthy recency attack. Two further baselines calibrate the space: the existing poisoning defenses benchmarked by Preference Poisoning Attacks on Reward Model Learning — Spectral Outlier Detection and the ALIBI differential-privacy defense — which that paper reports failed to consistently stop determined attackers in LLM safety domains; and our own ablations: gating removed (median-only, which isolates whether robust aggregation alone suffices) and median removed (gating-only, which isolates the drift-consistency ingredient).

Why not just…?

Why not just pool all data and train stationary DPO? Because the failure this proposal starts from is real: Right Now, Wrong Then shows stationary DPO collapses to roughly 50–55% reward accuracy when preferences flip, treating genuine drift as noise, and loses on downstream win rate (7.29 vs 8.93). Pooling buys robustness to our attack only by giving up drift adaptation entirely — and Chasing Moving Targets with Online Self-Play Reinforcement Learning for Safer Language Models separately shows static safety training mode-collapses onto obsolete threat patterns. Still, this alternative is a genuine threat to the proposal's relevance, and the experiments settle it directly: we will report the full accuracy-versus-robustness frontier of stationary DPO, and if stationary DPO's robustness advantage over our defense is large while its drift cost is small on realistic (gradual) drift schedules, the motivation for time-discounted learning — and hence for defending it — weakens. We consider this the most important negative result the study could produce.

Why not just run anomaly detection or differential privacy on the data stream? These are the obvious off-the-shelf defenses, and the literature is discouraging: Preference Poisoning Attacks on Reward Model Learning explicitly benchmarked Spectral Outlier Detection and ALIBI and both failed to consistently stop attackers, especially in LLM safety domains; PoisonBench shows poisoned models hold Stealthiness Scores above 98% on standard validation; and When Rubrics Backfire shows targeted drift can improve benchmark scores while degrading target-domain safety by up to 27.9%. The deeper problem is that outlier detection is exactly the wrong tool for our setting: recent data legitimately is distributionally unusual (that is what drift means), so a recency-aware outlier filter throws out the very signal discounting exists to capture. We will nonetheless run spectral filtering as a baseline under the recency attack, and our hypothesis — falsifiable here — is that it either misses the poison or deletes the drift.

Why not just pick a conservative discount factor $\gamma$ close to 1? This is the cheapest possible mitigation and deserves a straight answer: it works, partially. A $\gamma$ near 1 restores dilution by history and shrinks the attack, but it also destroys the drift adaptation that motivated discounting — the trade-off is continuous, not free. Our $\gamma$-sweep on NS-DPO under the stealthy recency attack will trace this frontier explicitly, and our defense's claim is precisely that it sits outside that frontier: at each $\gamma$ where NS-DPO achieves a given TV-HH drift accuracy, the gated variant should achieve comparable accuracy at substantially lower attack success. If the gated defense merely reproduces a point on the conservative-$\gamma$ frontier, the proposal's key ingredient adds nothing, and we will report that plainly.

Proposed Experiments

The experimental spine is the TV-HH dataset from Right Now, Wrong Then — a time-stamped split of Anthropic's HH-RLHF whose preference distribution genuinely shifts between helpfulness and harmlessness over time, giving us real benign drift to adapt to — overlaid with injected poison drawn from PKU-SafeRLHF-30k safety-flip labels (the setting where Preference Poisoning Attacks on Reward Model Learning reports a 100% attack success rate at 0.3% poisoning and 99% target-response selection at 5%) and with PoisonBench-style trigger backdoors effective at 0.1% poison. Poison budgets range from 0.05% to 5% of the stream. Primary metrics: clean-drift reward accuracy on held-out TV-HH time slices (does the model still track genuine drift, where NS-DPO's published reference is 60–70% under sudden flip and >10% margin over stationary DPO under gradual drift), attacker target-response selection rate after downstream policy training, and Stealthiness Score as defined in PoisonBench to confirm the attack evades benchmark validation, following the target-versus-benchmark evaluation design of When Rubrics Backfire.

Three checks are decisive. First, the $\gamma$-sweep: for each of stationary DPO, NS-DPO, and the gated defense, plot attacker success against clean drift accuracy across the full $\gamma$ range and poison budgets; the defense must dominate NS-DPO's frontier, not merely shift along it. Second, the timing experiment: hold the total poison budget fixed and vary only when it is injected; under timestamp-only weighting, success should scale as $\gamma^{-\Delta t}$ (our theory's signature prediction — if it does not, the theory is wrong and we say so). Third, the indistinguishability stress test: an adaptive adversary who explicitly optimizes poison to mimic the TV-HH drift direction, probing where the gate's covariate-shift estimate can be fooled; this locates the empirical analogue of our impossibility bound and bounds what any content-level defense can achieve. All comparisons use the same base model, identical data ordering, matched tuning budget, and identical poison realizations across methods, so differences are attributable to the weighting scheme alone.

The broader payoff is a clean answer to the tension nobody has yet addressed: systems that adapt to changing human values must decide which changes to follow, and the answer cannot come from timestamps. If the gated defense holds the frontier, temporal adaptation and adversarial robustness stop being a forced trade — and if it does not, we will have mapped exactly where that trade-off is fundamental.