Auditing Teammate Influence to Find and Remove Backdoors in Cooperative Multi-Agent Policies

Teams of reinforcement-learning agents, such as StarCraft unit squads, drone swarms, or robot fleets, learn to act by watching each other. That reliance is what makes them good at cooperating, and it is also a security hole. A teammate's behavior can serve as a hidden trigger. Imagine an agent policy trained so that it plays normally, but collapses for the rest of the episode once a teammate performs a specific short sequence of moves. We call this a backdoor: a training-time implant that stays dormant until a trigger appears. Today's defenses for reinforcement-learning backdoors look for triggers in what an agent senses, such as a pixel patch or a perturbed state. They do not look for triggers in what teammates do. Today's defenses against multi-agent attacks, meanwhile, rely on adversarial training, where the team trains against a learned attacker. That kind of training is expensive because every round needs fresh simulator rollouts against an evolving opponent.

This note proposes one cheap audit that addresses both problems. For each pair of agents, we measure how much one agent's action distribution changes when a teammate's recent actions are replaced with plausible alternatives. We then search for the short teammate action patterns that cause unusually large and persistent changes. In a clean team, influence between agents is spread out and fades quickly. In a backdoored team, a few specific motifs produce sharp spikes in influence that last the rest of the episode. Once these motifs are found, they can be removed by a constrained offline update that uses logged trajectories rather than an adversarial training loop. The broader bet is that backdoor triggers from teammates and coordinated test-time attacks such as the Wolfpack attack exploit the same weakness: influence concentrated on a few inter-agent channels. If so, one audit and one targeted repair should defend against both, at a small fraction of the cost of adversarial training.

Core Claim

We claim three things. First, teammate-triggered backdoors in cooperative multi-agent policies can be detected from logged play by finding outliers in counterfactual teammate influence. Second, they can be removed by an offline update applied only to the flagged channels, while preserving clean win rate. Third, the same targeted repair recovers a large share of the robustness that full adversarial training provides against coordinated attacks, at a fraction of the environment steps.

The first two claims are the core contribution. The third is a secondary hypothesis, and it could fail without sinking the first two.

Where This Sits in Prior Work

Three lines of work motivate the threat model.

BACKDOORL plants triggers made of opponent action sequences in recurrent policies for two-player competitive MuJoCo tasks. It shows that once triggered, the victim stays in a fast-failing mode for the whole episode. It also shows that fine-tuning on clean trajectories recovers win rate "only marginally."

Fox in the Henhouse (SCAB) shows that a third-party agent supplied to the training process can implant a backdoor by implicitly rewarding trigger responses. It reports a peak attack success rate above 90% with only 3% of training interactions corrupted.

CuDA2 shows that traitor agents inside a cooperative team can derail it through positioning and collisions alone.

All three establish that teammates and partners are a realistic channel for attacks. None of them proposes a defense for the cooperative, centrally trained setting where the trigger comes from a teammate.

Two existing defenses bound what is already solved.

BIRD reconstructs a trigger by optimizing a state perturbation that maximizes the agent's own value estimate. It then re-initializes the "shortcut" neurons and fine-tunes with clean rollouts. On SMAC it reaches detection F1 "often 1.0" against Marnet-style triggers. However, it treats each agent separately, searches over states rather than teammate actions, and assumes a clean environment.

SHINE explains failure episodes with a Gaussian-process attribution model and a learned feature mask. It then retrains in the possibly poisoned environment, reaching F1 above 0.93. However, it does not model triggers spread across cooperating agents, and its attribution model is expensive in large joint state spaces.

Our audit differs from both. It searches the space of teammate action motifs directly, and it repairs the policy offline.

Three papers supply method or contrast.

Wolfpack finds "helper" agents by measuring KL divergence between teammates' policies before and after an attack. Our influence score adapts that measurement, but uses it to audit rather than to attack. The paper's defense, WALL, trains against a Transformer-based planner and adds about 30% training time.

BHEA-AT cuts adversarial training time by 35–40% relative to ROMANCE and by over 60% relative to WALL. It does this by decoupling who and when to attack from how to hijack actions. It still alternates against a pool of adversaries that consumes rollouts.

MIR3 avoids adversaries entirely by penalizing the mutual information between an agent's history and its actions. Its authors note that this blanket bottleneck loses to explicit max-min training when the threat is narrow and specific. That limitation constrains our design. It is the reason we regularize only the channels the audit flags instead of every channel.

Finally, c-MBA learns a dynamics model of the multi-agent environment to compute attacks. We reuse that idea for a different purpose: rolling observations forward after a counterfactual change to a teammate's actions.

Research Program

Influence score. We define a counterfactual teammate-influence score for agent $i$ with respect to teammate $j$ at history $h$:

$$\Delta_{i\leftarrow j}(h) = \mathrm{KL}\big(\pi_i(\cdot \mid h)\,\|\,\pi_i(\cdot \mid \tilde h_{j})\big).$$

Here $\pi_i$ is agent $i$'s policy (the softmax over Q-values for QMIX agents). The counterfactual history $\tilde h_j$ is $h$ with teammate $j$'s last $k$ actions resampled from that teammate's own policy. The observations that follow are regenerated by a learned dynamics model, which keeps the counterfactual physically consistent rather than an impossible splice.

We also track how long the effect lasts. Persistence is the sum of $\Delta$ over the next $T$ steps after the window. This matters because recurrent backdoors keep the policy derailed long after the trigger ends, as BACKDOORL showed.

Trigger search. Detection is a search over discrete motifs of length $k$ for teammate $j$, or jointly for a pair of teammates when testing distributed triggers. The objective is to maximize persistent influence plus a drop in the centralized team value. We use the team value estimate from QMIX's mixing network, which plays the role BIRD gives to the single agent's value function.

We keep the search tractable in the way BHEA does for attacks. A cheap first stage chooses which teammate and which timestep to perturb, ranking them by $\Delta$ on logged data. A second stage optimizes the motif only for the top-ranked candidates, using beam search over actions. A policy is flagged when the best motif's persistent influence is an outlier relative to the influence distribution measured on held-out clean policies of the same architecture.

Repair. Repair is an offline objective on logged histories. Let $\pi_0$ be the suspect policy, $\mathcal{D}$ the logged histories, $h^{\mathrm{trig}}$ a history with a discovered motif inserted and rolled forward by the dynamics model, and $h^{\mathrm{ref}}$ the same history with that teammate's actions resampled instead. We train the repaired policy $\pi_\theta$ to minimize

$$\mathbb{E}_{h\sim\mathcal{D}}\,\mathrm{KL}\big(\pi_0(\cdot\mid h)\,\|\,\pi_\theta(\cdot\mid h)\big) + \lambda\,\mathbb{E}\,\mathrm{KL}\big(\mathrm{sg}[\pi_\theta(\cdot\mid h^{\mathrm{ref}})]\,\|\,\pi_\theta(\cdot\mid h^{\mathrm{trig}})\big),$$

where $\mathrm{sg}$ stops gradients and $\lambda$ trades off the two terms. The first term keeps clean behavior unchanged. The second makes the policy treat a triggered history the way it treats an ordinary one.

The update touches the policy weights, but it needs no new adversarial environment rollouts. The only environment interaction is a short validation phase at the end. We count every environment step and GPU-hour used, including logging, dynamics-model training, search, repair, and validation.

Coordinated attacks. For the secondary claim, we apply the same audit to clean but brittle policies. The top influence channels it finds should overlap with the "helpers" that Wolfpack targets. Repairing those channels with the second term above, using adversarial teammate actions as the inserted motif, should flatten the dependence that the Wolfpack attack exploits.

Baselines

The naive reference is vanilla QMIX with no defense. The existing figures for it are a drop from 97.6% to 11.4% win rate on SMAC 8m under Wolfpack (Wolfpack) and 10.2±3.5% on MMM under BHEA (BHEA-AT). For backdoors, the naive reference is the backdoored policy itself, plus that policy after plain fine-tuning on the same logged clean data our method uses.

Backdoor defenses. The strongest backdoor competitor is BIRD. It is the only method in the file that both detects and removes backdoors on SMAC, and it is reported at roughly twice the speed of the earlier Provable Defense. We run it as published: value-maximizing trigger restoration in observation space, shortcut-neuron re-initialization, and KL-constrained fine-tuning. We give it the clean environment access it assumes, which favors it.

We also run SHINE in the poisoned environment. All methods share the same suspect policies, the same logged data budget, and the same total budget of environment steps and GPU-hours. Each method's hyperparameters are tuned with an equal number of trials on a held-out map.

The reported results for both are against observation-space triggers on SMAC: BIRD F1 "often 1.0" and SHINE F1 above 0.93. For triggers from teammate actions or split across teammates, we found no reported result on Lacuna for either method.

Beating them means three things on teammate-triggered and distributed triggers. Detection F1 must be clearly higher. Triggered win rate must come back to within a few points of the clean policy. And total environment steps must be no greater than BIRD's.

Robustness baselines. For the secondary claim, the strongest competitors are BHEA-AT and WALL. Their reported numbers are 87.3±3.5% win rate on MMM under BHEA (BHEA-AT) and 83.6% on 8m under Wolfpack (WALL). We rerun both on our seeds rather than relying only on the quoted figures. Beating them means matching their attacked win rate within their reported spread while using substantially fewer environment steps than BHEA-AT. BHEA-AT is already the cheapest adversarial training method in the file.

Ablations. Two ablations isolate the key ingredient. The first applies MIR3's global mutual-information penalty at strengths matched so that its clean win rate equals ours. This tests whether targeting specific channels matters at all. The second removes the dynamics-model rollout and splices teammate actions without regenerating observations. This tests whether physically consistent counterfactuals matter.

Why not just…?

Why not just fine-tune the suspect policy on clean rollouts? BACKDOORL reports that clean fine-tuning barely recovers win rate against recurrent action-sequence backdoors. Fine-Pruning explains why: backdoor neurons stay dormant on clean data, so clean gradients never reach them. Our repair term differs because it trains on inputs that contain the trigger. We still include fine-tuning, and fine-pruning applied to the agents' recurrent layers, as baselines on identical data. If either recovers triggered win rate as well as our method, that alone would undercut the repair half of the proposal.

Why not just apply MIR3 everywhere? A blanket penalty on history-action information might suppress a trigger channel incidentally, and it costs almost nothing. This is the most serious alternative. MIR3's own authors note that it underperforms targeted max-min training against specific threats. A deterministic trigger reflex is also the kind of sharp dependence that may survive a penalty set weak enough to keep clean skill. The matched-clean-win-rate ablation settles this directly. If global MIR3 removes the backdoor at equal clean performance, our search step is unnecessary, and we would report that result.

Why not just extend BIRD's trigger search to actions? That is essentially part of what we do. The difference lies in what we optimize over and how we score it. We search teammates' discrete action motifs rather than continuous states. We score by persistent influence and team value rather than a single agent's value. And we keep counterfactual observations physically consistent with a dynamics model. The comparison with BIRD on distributed triggers, which need two teammates acting together, shows whether these choices matter. Searching over a single agent's actions cannot represent a trigger that only fires when two teammates act together.

Why not just run BHEA-AT and hope it covers backdoors too? Adversarial training against action hijacking explores perturbations chosen to cut reward. A specific dormant motif is unlikely to be among them unless the attacker stumbles onto it. We test this by evaluating BHEA-AT-trained policies that started from backdoored initializations. If they come out clean, the backdoor half of the claim weakens, but the cost comparison still stands.

Proposed Experiments

Building the attack. We first construct the missing attack on SMAC 8m and MMM with QMIX's recurrent agents. During centralized training, a designated teammate executes a length-4 to length-8 action motif in a small fraction of episodes, and the shaped reward favors the team's collapse afterward. This mirrors SCAB's implicit-reward mechanism and BIRD's Marnet setting. We build a single-teammate variant and a distributed variant in which two teammates must each perform half of the motif. The attack must reach a high attack success rate while keeping clean win rate near the 97.6% QMIX baseline on 8m. We also port the attack to MPE simple_spread and to PettingZoo Simple Push, so that the results are not specific to StarCraft.

Metrics. We report detection F1 over a population of clean and backdoored policies trained with different seeds, triggered and clean win rate after repair, total environment steps and GPU-hours, and for the secondary claim, win rate under the Wolfpack and BHEA attacks.

Decisive checks. There are four.

The first check is whether influence outliers separate backdoored from clean policies on distributed triggers, where BIRD's single-agent state search should fail. If they do not, the core claim is false.

The second is whether repair keeps the team within a few points of clean win rate while restoring triggered win rate. It must do so where clean fine-tuning and matched global MIR3 do not.

The third is an adaptive attacker. We retrain the backdoor with a penalty that keeps teammate influence below the detection threshold. This shows whether the attacker can evade the signal, or whether lowering the signal enough to evade detection also weakens the attack.

The fourth is the coordinated-attack transfer test. We compare the channels the audit flags against Wolfpack's helper sets. We then compare the attacked win rate after influence repair against the reported BHEA-AT and WALL numbers, and against the environment-step cost of each.