This paper, presented at NeurIPS 2026, tackles a fundamental brittleness in cooperative Multi-Agent Reinforcement Learning (MARL): the tendency for agents to overfit to specific training conditions and collapse when faced with "exogenous variability"—changes in factors outside their control, such as a shift in an opponent's strategy or a change in environmental physics.
TL;DR
The authors introduce LEICA, an algorithm that improves MARL generalization by explicitly modeling how agent actions indirectly influence the future behavior of uncontrollable environment components (like opponents). By partitioning the state space and rewarding actions that steer these "exogenous" elements, LEICA achieves significantly higher win rates on unseen task variations compared to standard baselines.
What Problem Does This Paper Address?
Current MARL algorithms, like MAPPO or QMIX, often achieve impressive results in static environments but fail when the "rules of the game" shift slightly. For example, a team of agents trained to defeat a specific StarCraft opponent might fail miserably if the opponent changes its targeting logic or starting distance.
The authors identify this as a failure to handle exogenous variability. These are factors that the agent team cannot directly control but which fundamentally alter the environment's transition dynamics. While some researchers try to solve this using "context inference" (trying to guess the current regime from history), the authors argue this isn't enough. As shown in Figure 1, even with context inference, there remains a massive gap between performance on "seen" training regimes and "unseen" test regimes.
Methods
The authors formalize the problem as an Exogenous Dec-POMDP (ED-POMDP). The core idea is to partition the global state $S_t$ into two parts:
- Directly action-responsive components ($S^d$): Things the agents' current actions change immediately (e.g., their own positions).
- Not directly action-responsive components ($S^r$): Things updated primarily by the opponent or environment (e.g., enemy health or positions).
Crucially, while agents can't change $S^r$ right now, they can exert mediated influence: their current actions change $S^d$, which in turn influences how $S^r$ evolves in the future.
LEICA (Learning Exogenous Influence for Coordination and Adaptation) implements this via:
- Context Encoders: Agents learn latent representations of the current regime from their local histories.
- Transition Predictors: The model learns to predict the next state for both $S^d$ and $S^r$.
- Influence-Weighted Rewards: LEICA calculates the Jacobian of the response predictor with respect to the directly responsive state. This measures how sensitive the future environment response is to the agents' immediate actions. Agents are given intrinsic rewards for taking actions that have high "mediated influence" on the environment's future state.
- Counterfactual Credit Assignment: To ensure individual agents are rewarded correctly, LEICA uses a counterfactual baseline to isolate one agent's contribution to the state change from the rest of the team.
Datasets
The primary evaluation is conducted on SMAX, a JAX-based implementation of the StarCraft Multi-Agent Challenge. The authors created a new evaluation protocol with 189 exogenous configurations based on five strategic axes (e.g., opponent targeting, movement mode, and distance).
- SMAX Scales: Small (3 train/3 test regimes), Medium (10/5), and Large (30/10).
- MABrax: A multi-agent continuous-control locomotion benchmark used to test physical perturbations (like wind).
- MPE-Tag: A mixed cooperative-competitive environment used for additional validation.
Key Insights and Ideas
- The "Aha" Moment: Recognizing that adaptation isn't just about identifying the regime, but about steering it. By rewarding agents for influencing the parts of the state they don't control, the model encourages behaviors that are robust across different opponent strategies.
- Jacobians as Influence Metrics: Using the gradient of the predicted environment response as a weight for intrinsic rewards is a clever way to quantify "influence" without needing a separate communication channel.
- Automatic Partitioning: While the state split can be done with domain knowledge, LEICA includes a module to automatically estimate which state variables are action-responsive by observing how they change under counterfactual action perturbations.
Key Results
LEICA consistently outperformed all baselines (MAPPO, QMIX, etc.) in generalization to unseen regimes.
- Overall Generalization: On the "Large" scale SMAX benchmark, LEICA achieved a 59.5% win rate on unseen opponents, compared to just 34.5% for standard MAPPO and 40.1% for MAPPO with context inference.
- Training Efficiency: LEICA often reached higher win rates faster than baselines, suggesting that the influence reward provides a helpful dense signal for coordination early in training.
- Physical Robustness: In MABrax experiments involving wind perturbations, LEICA showed superior returns on both seen and unseen wind dynamics compared to MAPPO.
Limitations and Future Work
- Noisy Transitions: The method relies on identifying action-dependent structures; very noisy environments or heavily hidden variables might degrade the quality of the influence signal.
- Fixed Opponents: The current study focuses on fixed (though reactive) opponents. Extending this to scenarios with learning opponents (co-adaptation) remains an open challenge.
- Computational Overhead: LEICA introduces about 13% more training-time computation due to the auxiliary predictors and Jacobian calculations, though execution remains decentralized and fast.
Summary updated