In single-agent reinforcement learning, an agent learns to navigate an environment with stationary transition dynamics. However, when multiple agents interact, the environment becomes inherently non-stationary from the perspective of any single agent. The optimal policy for one agent depends on the policies of its peers or adversaries, which are themselves continuously changing as they learn, adapt, or switch strategies.
Standard deep reinforcement learning algorithms often struggle in these scenarios because they treat other agents as static components of the environment. To overcome this "moving target" problem, an agent must anticipate and adapt to the evolving behaviors of others. Early approaches attempted to explicitly predict the exact next action of other agents. However, this is often too rigid and brittle; minor variations in execution can cause explicit action predictions to fail completely.
Instead of predicting immediate actions, a more robust paradigm has emerged: latent strategy inference. The core intuition is to observe an agent's historical behavior and compress it into a low-dimensional latent representation that captures their underlying style, intent, or policy type. By conditioning its own policy on this latent representation, an agent can effectively stabilize the environment and coordinate or compete more efficiently.
Tracking Behavior via Interaction History
How can an agent infer what another agent is planning to do without direct communication or access to their internal parameters? The answer lies in using historical interaction trajectories as a proxy for their hidden state.
In environments with minimal information, even an agent's own history can serve as a powerful signal. For instance, Jacob W. Crandall et al. demonstrated in Learning in Repeated Games with Minimal Information: The Effects of Learning Bias that when opponents' actions are completely unobserved, an agent can implicitly track opponent dynamics by incorporating its own previous actions and rewards into a recurrent state representation. The received reward acts as a proxy for the unobserved choices of the other agents, enabling coordination without explicit visibility.
When other agents' actions are observable, sequence models can build explicit behavioral profiles. Hal Daumé III et al. pioneered this implicit modeling approach with the Deep Reinforcement Opponent Network (DRON) in Opponent Modeling in Deep Reinforcement Learning, showing that learning a hidden representation of an opponent's style allows an agent to develop robust counter-strategies that generalize across minor variations in execution.
To capture these patterns over time, recurrent architectures are widely used to process historical trajectories:
- Tracking Strategy States: Chris Lu et al. in Model-Free Opponent Shaping used recurrent neural networks (RNNs) to process historical interaction trajectories as a proxy for an opponent's internal state, allowing a meta-learner to track and strategically influence opponent behavior in high-dimensional environments.
- Modeling Opponent Learning: Ian Davies et al. introduced LeMOL in Learning to Model Opponent Learning (Student Abstract), utilizing a bidirectional recurrent architecture to summarize an opponent's behavior over time. This captures not just their current strategy, but the trajectory of how they learn and adapt across an episode.
Learning Latent Representations via Auxiliary Tasks
To ensure that these latent representations capture meaningful strategic properties rather than noise, models are typically trained using an encoder-decoder architecture paired with predictive auxiliary tasks. The encoder maps the observed history to a low-dimensional latent space, while the decoder is forced to reconstruct or predict key environmental variables.
[ Historical Trajectory ] ---> ( Encoder ) ---> [ Latent Strategy Embedding ]
|
v
( Joint Policy / Value Network )
Different frameworks use different predictive targets to shape the latent space:
- Predicting Future Actions: In assistive robotics and human-AI collaboration, understanding the partner's policy is crucial. Jerry Zhi-Yang He et al. showed in Learning Representations that Enable Generalization in Assistive Tasks that predicting a partner's future actions serves as a highly effective objective for capturing their underlying policy, outperforming models that predict future states or rewards.
- Predicting States and Rewards: In Influencing Towards Stable Multi-Agent Interactions, Woodrow Zhouyuan Wang et al. proposed an encoder-decoder architecture where the decoder predicts future states and rewards conditioned on the current state, action, and inferred latent variable, allowing the agent to model the hidden policies of others.
- Reconstructing Trajectories under Partial Observability: When agents cannot see the full global state, modeling others becomes harder. In Agent Modelling under Partial Observability for Deep Reinforcement Learning, Georgios Papoudakis et al. introduced LIAM, which trains an encoder to map local history to a latent space while a decoder reconstructs the full trajectories of peers. Similarly, Dung Nguyen et al. in Social Motivation for Modelling Other Agents under Partial Observability in Decentralised Training utilized a dual-objective where the encoder-decoder reconstructs the agent's own state while simultaneously predicting the future actions of other agents.
- Joint Action and Policy Features: Shih-Yang Su et al. developed DPIQN in A Deep Policy Inference Q-Network for Multi-Agent Systems, where an auxiliary module learns policy features directly from raw observations to generate latent representations of spatial-temporal behavioral patterns.
To prevent the latent space from overfitting or memorizing irrelevant details, regularization techniques like the variational information bottleneck can be applied. Lei Yuan et al. leveraged this in Multi-Agent Incentive Communication via Decentralized Teammate Modeling, constructing compact latent representations of teammate behavior from local observations to anticipate peer actions and generate tailored incentive messages.
Generalizing to Unknown Partners (Ad Hoc Teamwork)
One of the most powerful applications of latent strategy inference is ad hoc teamwork, where an agent must collaborate with unfamiliar teammates without prior coordination or shared training.
By training temporal models on a diverse population of training agents, the learned encoder can generalize to entirely unseen behaviors by mapping them to the correct region of the latent space. Arrasy Rahman et al. developed Graph-based Policy Learning (GPL) in Towards Open Ad Hoc Teamwork Using Graph-based Policy Learning and extended it in A General Learning Framework for Open Ad Hoc Teamwork Using Graph-based Policy Learning, demonstrating that processing historical teammate actions into latent embeddings allows a learner to identify underlying policy patterns and adapt to unknown teammates in real-time.
When dealing with a mixture of controlled and uncontrolled agents, Caroline Wang et al. proposed POAM in N-agent Ad Hoc Teamwork. POAM uses an encoder-decoder architecture to profile teammates on the fly by predicting future actions and observations, enabling real-time adaptation to diverse coordination styles.
Furthermore, when agents encounter human or AI teammates sequentially over time, they must adapt without forgetting previous partners. Cong Guan et al. addressed this in One by One, Continual Coordinating with Humans via Hyper-Teammate Identification, using a recurrent module for few-shot profiling that encodes a teammate's behavioral patterns into a latent embedding, allowing rapid identification of a partner's strategy and preferences without extensive prior interaction.
Long-Term Behavioral Steering and Strategy Shaping
Latent strategy inference does not just enable passive adaptation; it also allows agents to proactively shape or steer the behavior of others. If an agent understands how its actions influence the latent strategy of its partner, it can choose actions that guide the partner toward a more optimal, stable behavior.
In autonomous driving, Xiyang Wu et al. introduced iPLAN in Intent-Aware Planning in Heterogeneous Traffic via Distributed Multi-Agent Reinforcement Learning, using a recurrent encoder-decoder to infer latent behavioral traits (such as driving styles) from historical observations, allowing autonomous vehicles to plan safe paths and interact smoothly with heterogeneous traffic.
When interacting with humans, an agent can use these models for long-term behavioral steering. Joey Hong et al. explored this in Learning to Influence Human Behavior with Offline Reinforcement Learning. By representing human behavior as a latent strategy within a hidden state model (such as a Hidden Markov Decision Process), the agent can predict how its own actions will pressure a human partner to transition between different underlying strategies. Instead of merely reacting to immediate human actions, the agent applies environmental constraints that guide the human toward more effective long-term coordination tactics.
By treating the interaction history as a window into the hidden minds of other agents, latent strategy inference transforms multi-agent reinforcement learning from a chaotic, non-stationary challenge into a predictable, structured environment where coordination and strategic influence become possible.
Key Foundational Work
Learning in Repeated Games with Minimal Information: The Effects of Learning Bias (2011) — contributed 2 concept elements to this direction.
Opponent Modeling in Deep Reinforcement Learning (2016) — contributed 1 concept elements to this direction.