Many deployed learning agents — bidding bots, trading systems, game-playing policies — keep learning while they are used, updating their behavior from the payoffs they receive in live interactions. A separate line of work on backdoor attacks shows that learning systems can carry hidden, trigger-activated failure modes: the model behaves normally until it sees an attacker-chosen cue, then fails in a way the attacker profits from. The uncomfortable observation motivating this proposal is that these two threats have never met on their natural common ground. Every known backdoor against a learning agent requires access to the training pipeline — poisoned datasets, rewritable replay buffers, or controlled model code. And every known way of manipulating an online learner through interaction requires the attacker to keep working: it must continuously play sacrificial decoy moves to hold the victim in an exploitable state. Neither literature has asked whether an opponent who controls nothing but its own moves in the game can plant a dormant backdoor in an online learner — one that costs nothing to maintain, activates on a sparse trigger, and erases itself afterward.
A mean-based learner, in the sense of Strategizing against No-regret Learners, picks its next action by comparing the historical average payoff of each of its actions; multiplicative-weights and follow-the-perturbed-leader are the canonical examples, and Q-learners behave analogously with a value table in place of an average. That paper proved an optimizer can inflate a chosen action's historical average by playing sub-optimal decoys, locking a mean-based learner with three or more actions into a predictable move and extracting more than the Stackelberg value $V$ — but the steering is continuous and the learner's policy stays visibly distorted. We propose to replace continuous steering with a two-phase implant-then-harvest protocol, and to characterize exactly when an online learner's own memory dynamics make it possible.
Core Claim
The claim has three parts. First, feasibility: any mean-based learner whose memory is recency-weighted — exponentially discounted averages, sliding windows, or standard Q-value updates — admits a trigger-activated backdoor planted purely through the attacker's action choices. The attacker first plays normally (the dormancy phase), then executes a short sequence of in-distribution actions whose effect is to make the victim's recent average payoff for a chosen victim action dominate its alternatives; the victim locks onto that action for roughly one memory horizon; the attacker exploits; the victim's window then refills with honest payoffs and the backdoor erases. Second, stealth under the victim's own guarantee: the attacker's total extraction above $V$ is paid for out of the victim's regret, and no-regret learners only promise that regret is small, not zero:
$$U_{\text{attacker}} \;\le\; V + R_T, \qquad R_T = O(\sqrt{T \log N}),$$
where $R_T$ is the victim's cumulative regret over $T$ rounds against $N$ actions (sublinear for standard mean-based algorithms, per the setting of Strategizing against No-regret Learners). Sparse attack bursts therefore fit entirely under the victim's advertised performance guarantee, so regret monitoring — the natural health check for an online learner — certifies the victim as healthy while it is being farmed. Third, boundary: with exact uniform averaging, a trigger round's weight decays like $1/T$, so no persistent backdoor exists and only shrinking transients are exploitable; the attack surface is precisely recency-weighted memory. This formalizes the open tension between averaging-based dilution and trigger persistence that the file identifies as unanalyzed.
Research Program
The first workstream formalizes the threat model and the attack. The adversary chooses only its own actions in a repeated bimatrix or Markov game; it cannot touch the victim's rewards, code, or data. The implantation problem — finding a trigger sequence that maximizes post-trigger extraction subject to a stealth constraint (the trigger's action distribution must stay statistically close to normal play) — is a path-finding control problem in the victim's memory state, directly analogous to the $(N-1)$-dimensional search posited but not solved in Strategizing against No-regret Learners. We will compute trigger sequences with the quantized dynamic-programming machinery of Strategizing Against Q-Learners: A Control-Theoretical Approach, which already solves closed-loop control over a discretized Q-value state space, but with a modified objective: maximize discounted extraction over sparse attack cycles while penalizing deviation from the normal-play distribution, instead of maximizing steady-state utility. Their formulation assumes full knowledge of the victim's algorithm and parameters; we keep that assumption for the theory and treat limited-knowledge attackers (system identification from observed play) as a defined stretch goal, since the file flags it as open.
The second workstream is the theory: a dilution theorem separating erasing from persistent backdoors (uniform averaging: transient only; discounted memory: self-erasing backdoors with recovery time $\approx 1/(1-\gamma)$; Q-learners whose state includes the opponent's last action: genuinely persistent conditional mappings, since "adversary played trigger → victim plays target" is stored in the value table rather than in a fading window); the regret-budget inequality above; and an extension of the no-swap-regret defense theorem of Strategizing against No-regret Learners to the triggered regime — our hypothesis is that swap-regret-minimizing victims cap attacker utility at $V$ here too, making no-swap regret the principled defense against behavioral backdoors, a connection no paper in the file tests.
The third workstream evaluates defenses. The existing RL backdoor defenses — neuron pruning plus fine-tuning Fine-Pruning: Defending Against Backdooring Attacks on Deep Neural Networks, entropy probing under input blending STRIP: A Defence Against Trojan Attacks on Deep Neural Networks, and SVD-based subspace sanitization Provable Defense against Backdoor Policies in Reinforcement Learning — all require guaranteed clean calibration data and assume triggers are input-space artifacts orthogonal to task features. Our triggers are valid, in-distribution actions by an opponent inside the game's natural dynamics, the same property that makes behavioral triggers in BACKDOORL: Backdoor Attack against Competitive Reinforcement Learning resistant to fine-tuning, so we expect these defenses to be inapplicable or ineffective; we will test the strongest applicable ones rather than assert this.
Baselines
The do-nothing reference is self-play: two standard independent Q-learners in the repeated Prisoner's Dilemma reach mutual cooperation at approximately 95% frequency and a second learner acting as the "attacker" earns about 0.10 utility, as reported in Strategizing Against Q-Learners: A Control-Theoretical Approach. Any real attack must beat this while leaving the victim's long-run average utility close to its self-play level; if the victim's utility collapses, the attack is not a backdoor but obvious exploitation. The strongest existing competitor is the continuous control-theoretic steering attack from the same paper, which raised the strategic actor's Prisoner's Dilemma utility from approximately 0.10 to 0.20 while dropping mutual cooperation to roughly 60%, and achieved utility 0.5340 against 0.4465 for a standard Q-learner in randomly generated zero-sum games. We will run it under identical conditions — same games, same victim algorithms and hyperparameters, same total rounds, and the same quantized dynamic-programming solver with the same tuning budget — so the only difference is the attack policy. Beating it means matching or exceeding its attacker utility while spending at least an order of magnitude fewer rounds on off-equilibrium steering moves and keeping the victim's realized regret within its $O(\sqrt{T \log N})$ guarantee, which continuous steering plausibly violates by holding the victim's policy persistently away from its best response. A naive greedy attacker that simply plays its one-shot best response every round is included as a floor. Finally, an ablation isolates the proposal's key ingredient, the victim's memory model: the identical implantation protocol is run against undiscounted cumulative multiplicative weights, exponentially discounted multiplicative weights, and windowed follow-the-perturbed-leader, with only the memory rule changed; the dilution theorem predicts the attack's success rate collapses for the cumulative variant.
Why not just…?
Why not just continuously steer the victim, as prior work already does? This is the most serious objection, because continuous steering is proven to work: Strategizing Against Q-Learners: A Control-Theoretical Approach already extracts 0.20 utility in the Prisoner's Dilemma and 0.5340 in random zero-sum games, and Strategizing against No-regret Learners proves the general mean-based case. The honest answer is that on raw utility per round, continuous steering may match or beat us, and our head-to-head experiment is designed to concede that possibility. The claim we defend instead is about cost and detectability: continuous steering requires permanent sacrifice of short-term utility on decoys and holds the victim's policy visibly off-equilibrium, whereas a dormant backdoor has zero ongoing steering cost and leaves the victim's regret inside its own guarantee. The experiments measure utility per round of deviating play and detectability by a regret monitor, and if continuous steering wins on both, the proposal's practical contribution collapses to the theory — a risk we state up front.
Why not just poison the training pipeline like existing backdoor work? Because that is a different, stronger attacker. SleeperNets achieves a 100% attack success rate at poisoning rates as low as 0.0006%, but only with write access to the replay buffer; BACKDOORL requires a malicious developer controlling the training loss and architecture; Adversarial Inception Backdoor Attacks against Reinforcement Learning needs buffer access to swap actions in stored transitions. An agent that learns online from live opponents — the setting the manipulation literature actually studies — has no pipeline an outsider can poison. Our question is whether the weaker, purely interactive attacker can still plant a backdoor; the answer delimits which deployed systems are exposed to anyone they happen to play against, not just to their developers.
Why not just detect the attack with anomaly monitoring or purge it by retraining? The literature says the standard tools are misaligned with this threat. Static reward-shaping attacks do fail in constrained settings — SleeperNets shows static poisoning stagnating between 0% and 90% attack success in Highway Merge, and Adversarial Inception shows it collapsing to near 0% under $[0,1]$ reward clipping — but our attack writes no rewards at all, so reward outlier detection has nothing to see; the attacker's payoffs to the victim are the game's own. Fine-tuning on clean data only marginally improves the victim against behavioral backdoors, as BACKDOORL reports for its entrenched LSTM policies, and Fine-Pruning shows plain fine-tuning fails even in vision because backdoor circuits stay dormant on clean data. Subspace sanitization Provable Defense against Backdoor Policies in Reinforcement Learning assumes triggers orthogonal to task features in observation space; an opponent's legitimate action lies inside the task space by construction. The one monitoring signal that could in principle work — tracking the victim's realized regret — is precisely the signal our regret-budget analysis predicts will stay quiet, and we test this directly.
Why not just have the victim use a no-swap-regret learner? This is a real threat to the attack's practical importance, not a straw man: Strategizing against No-regret Learners proves no-swap-regret algorithms cap a continuous manipulator at the Stackelberg value $V$, and if the same holds in our triggered regime, a sophisticated victim simply opts out. We expect it largely does hold — and we treat proving and empirically confirming this as a deliverable, not a liability, because it converts the proposal from "new attack" into "characterization: every standard mean-based learner deployed today is exposed to purely interactive backdoors, and swap-regret minimization is the matching defense." The honest caveat is that swap-regret learners are rarely what gets deployed; the vulnerability statement is about the world as it is.
Proposed Experiments and Decisive Checks
The core experiments run in the repeated-game settings the manipulation literature already uses: the repeated Prisoner's Dilemma and randomly generated zero-sum and general-sum bimatrix games with $N$ from 3 to 10 actions (three or more is where beating $V$ becomes possible, per the phase-shift theorems of Strategizing against No-regret Learners). Victims are multiplicative weights, follow-the-perturbed-leader, their discounted and windowed variants, independent Q-learners, and a swap-regret learner; the attacker is our implantation protocol computed via the quantized dynamic programming of Strategizing Against Q-Learners. Metrics are attack success rate (fraction of cycles in which the victim plays the target action within its memory horizon after the trigger), attacker utility per cycle, steering cost (fraction of rounds the attacker deviates from its normal-play distribution), victim recovery time, the victim's realized regret against its $O(\sqrt{T \log N})$ guarantee, and a stealth score given by a two-sample test distinguishing the attacker's action distribution in attack epochs from dormancy epochs. Concrete targets: in the Prisoner's Dilemma, attacker utility at or above the 0.20 continuous-steering result with steering cost below 10% of rounds; in random zero-sum games, utility at or above the reported 0.5340 under the same conditions. For the persistent variant, we measure how long the conditional mapping "trigger → target action" survives in the Q-learner's table without re-triggering, contrasting it with the predicted self-erasure under windowed averaging. The decisive checks are three: the memory ablation (if the cumulative-average victim is attacked as successfully as the discounted one, the dilution theory is wrong and the proposal's mechanism story fails); the regret monitor (if monitoring realized regret flags our sparse bursts, the stealth claim fails); and the swap-regret victim (if the attacker exceeds $V$ there, the proposed defense fails and the vulnerability is broader than the theory of Strategizing against No-regret Learners predicts). As a stretch experiment extending toward the deep-RL regime, we apply the protocol against a deep policy trained online in the competitive MuJoCo environments used by BACKDOORL — Run to Goal and Sumo — with the bar being a victim win-rate drop comparable to the 17% to 37% that attack achieves, but without any access to the training pipeline that BACKDOORL's malicious developer enjoys.
References
- Strategizing against No-regret Learners
- Strategizing against Q-learners: A Control-theoretical Approach
- Fine-Pruning: Defending Against Backdooring Attacks on Deep Neural Networks
- STRIP: a defence against trojan attacks on deep neural networks
- Provable Defense against Backdoor Policies in Reinforcement Learning
- BACKDOORL: Backdoor Attack against Competitive Reinforcement Learning
- SleeperNets: Universal Backdoor Poisoning Attacks Against Reinforcement Learning Agents
- Adversarial Inception Backdoor Attacks against Reinforcement Learning