In multi-agent reinforcement learning, we often assume that agents adapt to one another until they reach a stable equilibrium, such as a Nash or Stackelberg equilibrium. However, when an opponent uses a predictable learning rule, a strategic agent can abandon standard equilibrium play entirely. Instead of treating the opponent as an active adversary, the strategic agent can treat the opponent's learning algorithm as a deterministic, controllable dynamical system.
By understanding how the learner updates its internal state based on environmental feedback, a strategic actor can feed specific, biased data into the game to steer the learner toward a desired behavior. This shift from reacting to steering allows strategic agents to extract rewards far beyond traditional game-theoretic benchmarks.
Learning as a Control Problem
Why are adaptive agents so easy to manipulate? The vulnerability stems from a fundamental blind spot: most learning algorithms assume their environment is either stationary or changing independently of their own internal updates. They do not expect an opponent to actively read their internal state and calculate moves designed to alter their future behavior.
As Yuksel Arslantas et al. demonstrate in Strategizing Against Q-Learners: A Control-Theoretical Approach, if an opponent's internal update rules (such as a Q-function) are predictable, their learning process becomes an extension of the environment's transition dynamics. By treating this internal state as part of a closed-loop control problem, a strategic actor can sequence its actions to guide the learner's policy toward a state that maximizes the actor's long-term utility.
This insight traces back to early observations by Gerald Tesauro in Extending Q-Learning to General Adaptive Multi-Agent Systems, who demonstrated that modeling an adversary's adaptation cycles allows an agent to exploit transient policy shifts and systematic mistakes rather than settling for a defensive, lower-value Nash equilibrium.
The Vulnerability of Mean-Based Learning
To build intuition for how this works, consider a common class of online learning algorithms known as mean-based or no-regret learners. These algorithms prioritize actions based on their cumulative historical performance, assuming that past success predicts future rewards.
This reliance on historical averages creates a massive loophole. In Strategizing against No-regret Learners, Yuan Deng et al. explain the mechanism of strategic payoff shaping. A strategic player can intentionally play sub-optimal actions in the short term to artificially inflate the historical average of a specific opponent action. The mean-based learner, tracking these averages, sees a highly lucrative option and converges onto that predictable move. Once the learner is locked in, the strategic player shifts to a counter-strategy that ruthlessly exploits it.
[Strategic Agent] ---> Plays Suboptimal Decoys ---> Inflates Historical Averages
|
v
[Strategic Agent] <--- Exploits Locked-in Move <--- [Mean-Based Learner Converges]
As William Brown et al. argue in Is Learning in Games Good for the Learners?, this vulnerability means that standard learning rules often leave money on the table. Adversaries can manipulate the input history over time—a risk that Jon Schneider et al. also identify in auction settings in Strategically-Robust Learning Algorithms for Bidding in First-Price Auctions—allowing exploiters to achieve rewards that far exceed standard game-theoretic benchmarks.
Reversing the Stackelberg Advantage
This vulnerability completely upends traditional power dynamics in sequential, leader-follower (Stackelberg) games. In a classic Stackelberg game, the leader commits to a strategy first, and the follower responds. Because the leader can anticipate the follower's best response, the leader holds a structural first-mover advantage.
However, if a leader attempts to learn the follower's preferences through repeated interactions, the follower can exploit this learning phase to reverse the advantage. In Optimally Deceiving a Learning Leader in Stackelberg Games, Georgios Birmpas et al. show that a follower can systematically misrepresent their preferences by providing deceptive responses that correspond to a fake payoff matrix. By doing so, the follower tricks the leader into making a commitment that serves the follower's hidden objectives.
The authors of Imitative Attacker Deception in Stackelberg Security Games reveal a highly effective version of this tactic: the follower can pretend to have a strictly competitive, zero-sum payoff structure. This imitative deception forces the learning leader to retreat to a conservative "maximin" safety strategy. Once the leader is locked into this predictable baseline, the follower can easily optimize against it.
As Jiarui Gan et al. note in Manipulating a Learning Defender and Ways to Counteract, this manipulation renders the leader's learning phase entirely redundant. Through this mechanism, followers with side information can effectively "train" the leader, securing a massive reward advantage over honest best-responses (Yaolong Yu et al. in Decentralized Online Learning in General-Sum Stackelberg Games; Yurong Chen et al. in Optimal Private Payoff Manipulation Against Commitment in Extensive-form Games).
The Decoy and Exploit Cycle
To successfully execute this control, strategic agents typically rely on a multi-phase behavioral loop. They cannot simply lie once; they must manage the opponent's learning trajectory over time.
In Playing is believing: The role of beliefs in multi-agent learning, Yu-Han Chang et al. formalize this via an alternating cycle of decoy and exploitative policies. The strategic agent intentionally accepts small, short-term losses under a decoy policy to lure the opponent into a specific, predictable adaptation path. Once the opponent's strategy has converged to the desired state, the agent rapidly shifts to an exploitative policy designed to counter that exact response.
This temporal manipulation allows agents to guide the long-term co-adaptation of their partners. As Annie Xie et al. demonstrate in Learning Latent Representations to Influence Multi-Agent Interaction, agents can proactively concede minor advantages to guide non-stationary opponents into predictable patterns.
Whether by projecting false affective signals (Nabiha Asghar et al. in Intelligent Affect: Rational Decision Making for Socially Aligned Agents), misreporting internal constraints to signal rigid commitments (Noam Nisan et al. in How and Why to Manipulate Your Own Agent: On the Incentives of Users of Learning Agents), or explicitly modeling opponent behavior to force them into disadvantageous positions (Bilal Kartal in Agent Modeling as Auxiliary Task for Deep Reinforcement Learning), the core intuition remains identical: a learning agent is not a fixed target, but a dynamic system waiting to be steered.
Key Foundational Work
Playing is believing: The role of beliefs in multi-agent learning (2001) — contributed 4 concept elements to this direction.
Advantage Updating Applied to a Differential Game (1994) — contributed 1 concept elements to this direction.