Addressing Exogenous Variability in Cooperative Multi-Agent Reinforcement Learning

Seongmin Kim, Woohyeon Byeon, Jiwon Jeon, Seungyul Han, Youngchul SungarXiv 2026 · arXiv

This paper, presented at NeurIPS 2026, tackles a fundamental brittleness in cooperative Multi-Agent Reinforcement Learning (MARL): the tendency for agents to overfit to specific training conditions and collapse when faced with "exogenous variability"—changes in factors outside their control, such as a shift in an opponent's strategy or a change in environmental physics.

TL;DR

The authors introduce LEICA, an algorithm that improves MARL generalization by explicitly modeling how agent actions indirectly influence the future behavior of uncontrollable environment components (like opponents). By partitioning the state space and rewarding actions that steer these "exogenous" elements, LEICA achieves significantly higher win rates on unseen task variations compared to standard baselines.

What Problem Does This Paper Address?

Current MARL algorithms, like MAPPO or QMIX, often achieve impressive results in static environments but fail when the "rules of the game" shift slightly. For example, a team of agents trained to defeat a specific StarCraft opponent might fail miserably if the opponent changes its targeting logic or starting distance.

The authors identify this as a failure to handle exogenous variability. These are factors that the agent team cannot directly control but which fundamentally alter the environment's transition dynamics. While some researchers try to solve this using "context inference" (trying to guess the current regime from history), the authors argue this isn't enough. As shown in Figure 1, even with context inference, there remains a massive gap between performance on "seen" training regimes and "unseen" test regimes.

Figure 1: Motivation. (a) An episode-level exogenous regime (e.g., opponent strategy and initial distance) changes how a fixed but state-reactive opponent responds. Ally actions directly change ally state (e.g., positions), which then affects the opponent’s later movement and attacks. (b) Policies trained against the default SMAX opponent achieve high performance but collapse under unseen opponent strategies. (c) Multi-regime training with VariBAD-style context inference [39] (MAPPO+ctx) still suffers under exogenous variation.

Methods

The authors formalize the problem as an Exogenous Dec-POMDP (ED-POMDP). The core idea is to partition the global state $S_t$ into two parts:

  1. Directly action-responsive components ($S^d$): Things the agents' current actions change immediately (e.g., their own positions).
  2. Not directly action-responsive components ($S^r$): Things updated primarily by the opponent or environment (e.g., enemy health or positions).

Crucially, while agents can't change $S^r$ right now, they can exert mediated influence: their current actions change $S^d$, which in turn influences how $S^r$ evolves in the future.

Figure 2: Illustration of delayed mediated influence. Actions at time $t$ influence the responsive state $S^d_{t+1}$, which then steers the opponent's response $S^r_{t+2}$.

LEICA (Learning Exogenous Influence for Coordination and Adaptation) implements this via:

  • Context Encoders: Agents learn latent representations of the current regime from their local histories.
  • Transition Predictors: The model learns to predict the next state for both $S^d$ and $S^r$.
  • Influence-Weighted Rewards: LEICA calculates the Jacobian of the response predictor with respect to the directly responsive state. This measures how sensitive the future environment response is to the agents' immediate actions. Agents are given intrinsic rewards for taking actions that have high "mediated influence" on the environment's future state.
  • Counterfactual Credit Assignment: To ensure individual agents are rewarded correctly, LEICA uses a counterfactual baseline to isolate one agent's contribution to the state change from the rest of the team.

Datasets

The primary evaluation is conducted on SMAX, a JAX-based implementation of the StarCraft Multi-Agent Challenge. The authors created a new evaluation protocol with 189 exogenous configurations based on five strategic axes (e.g., opponent targeting, movement mode, and distance).

  • SMAX Scales: Small (3 train/3 test regimes), Medium (10/5), and Large (30/10).
  • MABrax: A multi-agent continuous-control locomotion benchmark used to test physical perturbations (like wind).
  • MPE-Tag: A mixed cooperative-competitive environment used for additional validation.

Key Insights and Ideas

  • The "Aha" Moment: Recognizing that adaptation isn't just about identifying the regime, but about steering it. By rewarding agents for influencing the parts of the state they don't control, the model encourages behaviors that are robust across different opponent strategies.
  • Jacobians as Influence Metrics: Using the gradient of the predicted environment response as a weight for intrinsic rewards is a clever way to quantify "influence" without needing a separate communication channel.
  • Automatic Partitioning: While the state split can be done with domain knowledge, LEICA includes a module to automatically estimate which state variables are action-responsive by observing how they change under counterfactual action perturbations.

Key Results

LEICA consistently outperformed all baselines (MAPPO, QMIX, etc.) in generalization to unseen regimes.

  • Overall Generalization: On the "Large" scale SMAX benchmark, LEICA achieved a 59.5% win rate on unseen opponents, compared to just 34.5% for standard MAPPO and 40.1% for MAPPO with context inference.
  • Training Efficiency: LEICA often reached higher win rates faster than baselines, suggesting that the influence reward provides a helpful dense signal for coordination early in training.
  • Physical Robustness: In MABrax experiments involving wind perturbations, LEICA showed superior returns on both seen and unseen wind dynamics compared to MAPPO.

Limitations and Future Work

  • Noisy Transitions: The method relies on identifying action-dependent structures; very noisy environments or heavily hidden variables might degrade the quality of the influence signal.
  • Fixed Opponents: The current study focuses on fixed (though reactive) opponents. Extending this to scenarios with learning opponents (co-adaptation) remains an open challenge.
  • Computational Overhead: LEICA introduces about 13% more training-time computation due to the auxiliary predictors and Jacobian calculations, though execution remains decentralized and fast.

Summary updated

Abstract

Cooperative multi-agent reinforcement learning (MARL) often fails under exogenous variability such as shifts in opponents or environment regimes that cannot be controlled by the team but alter the transition dynamics. We formalize this challenge as Exogenous Dec-POMDP (ED-POMDP), in which an episode-level exogenous regime, selected independently of the team, indexes the dynamics, and the global state is partitioned into components that are directly responsive to the current team action and components that are not directly action-responsive at the current step. This formulation exposes mediated influence: while team actions do not directly update the latter at the current step, they can shape their later evolution through induced changes in the former. Based on this, we propose LEICA, a CTDE-compatible algorithm that learns history-conditioned context representations for both components and shapes policy updates using agent-wise, influence-weighted counterfactual intrinsic rewards. Across SMAX benchmarks with opponent strategy shifts, LEICA consistently improves both training performance and generalization to unseen opponents' strategies over existing baselines, supporting the usefulness of influence-weighted shaping under exogenous train-test regime shifts.
BibTeX
@misc{kim2026addressing,
  author = {Seongmin Kim and Woohyeon Byeon and Jiwon Jeon and Seungyul Han and Youngchul Sung},
  title = {{Addressing Exogenous Variability in Cooperative Multi-Agent Reinforcement Learning}},
  year = {2026},
  eprint = {2611.22067},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.22067},
}
Figure