ERMAS: Becoming Robust to Reward Function Sim-to-Real Gaps in Multi-Agent Simulations

Eric Zhao, Alexander R. Trott, Caiming Xiong, Stephan Zheng

Abstract

Robust decision-making in multiplayer games requires anticipating what reactions a player policy may elicit from other players. This is difficult in games with three or more players: when one player fixes a policy, it induces a subgame between the remaining players with many possible equilibria outcomes. Predicting the worst-case outcome of a policy is thus an equilibrium selection problem -- one known to be generally NP-Hard. We show that worst-case coarse-correlated equilibria can be efficiently approximated in smooth games and propose a framework that uses the worst-case evaluation scheme to learn robust player policies. We further prove the framework can be extended to handle uncertainty about the bounded rationality of other players. In experiments, our framework learns robust policies in repeated N-player matrix games and, when applied to deep multi-agent reinforcement learning, can scale to complex spatiotemporal games. For example, it learns robust AI tax policies that improve welfare by up to 15%, even when taxpayers are boundedly rational.

Introduction: The Challenge of Strategic Robustness

In multi-agent reinforcement learning (MARL), we often train a "principal" agent—such as a government or platform designer—under the assumption that other agents (the "followers") will respond with perfect rationality. However, this approach faces two significant hurdles in real-world application: equilibrium selection and bounded rationality.

A single policy may lead to multiple stable outcomes (equilibria). If a principal trains against an optimistic response, the policy may fail if real-world agents coordinate on a less favorable outcome. Furthermore, real humans are not perfect utility-maximizing agents; they exhibit bounded rationality, making mistakes or pursuing goals that differ slightly from simulation assumptions. Standard methods for identifying the "worst-case" response to a policy are computationally intractable (NP-hard), creating a significant barrier to achieving strategic robustness.

Methodology: Sampling Worst-Case Equilibria

The ERMAS framework addresses these challenges by evaluating a principal's policy against the worst-case Coarse-Correlated Equilibrium (CCE). The core theoretical breakthrough lies in the observation that for "smooth games"—where an individual's impact on the total system is bounded—finding the worst-case CCE is tractable in polynomial time using no-regret dynamics.

Conceptual overview of the ERMAS framework, illustrating how it samples worst-case boundedly rational behaviors to bridge the sim-to-real gap.

To implement this practically, we utilize a weighting system involving Lagrange multipliers. Each agent's objective is modified to balance their own reward against an adversarial objective toward the principal. By tuning a "reward slack" parameter ($\epsilon$), the principal can train against agents that are mostly rational but willing to deviate slightly to exploit the principal's weaknesses. This allows the framework to sample from a broader set of behaviors, including those that are boundedly rational, rather than just the naively sampled behaviors found in standard simulations.

Key Results: Robustness in Complex Economies

The framework was evaluated across several environments, including sequential bimatrix games, N-agent matrix games, and the high-fidelity AI Economist simulation. In the AI Economist, the principal acts as a tax planner for four agents in a spatiotemporal trade-and-barter economy.

Performance of ERMAS against various taxpayer models in the AI Economist simulation, demonstrating superior robustness compared to standard baselines.

Key findings include:

  • Economic Impact: In the AI Economist simulation, the robust tax policy improved social welfare (a metric balancing equality and productivity) by up to 15% compared to standard baselines when facing previously unseen, boundedly rational taxpayers.
  • Generalization: In matrix games, principals trained with ERMAS performed within the standard error of top-performing models across all agent types (Vanilla, Adversarial, and Risk-Averse). In contrast, standard MARL models failed significantly when the agent type shifted at test time.
  • Closing the Robustness Gap: ERMAS successfully closed the gap where complex dynamic RL policies often underperform compared to simple static formulas (like the Saez formula) when faced with unpredictable agent behavior.

Conclusion and Future Work

ERMAS demonstrates that strategic robustness is a requirement for successful sim-to-real transfer in social systems. While training against adversarial agents is more computationally intensive than standard MARL, the gains in reliability and performance are substantial. Future research will focus on building uncertainty sets based on historical human data to create even more realistic adversarial models and improving the convergence sensitivity of the adversarial sampling algorithm.

Summary updated

Abstract

Robust decision-making in multiplayer games requires anticipating what reactions a player policy may elicit from other players. This is difficult in games with three or more players: when one player fixes a policy, it induces a subgame between the remaining players with many possible equilibria outcomes. Predicting the worst-case outcome of a policy is thus an equilibrium selection problem -- one known to be generally NP-Hard. We show that worst-case coarse-correlated equilibria can be efficiently approximated in smooth games and propose a framework that uses the worst-case evaluation scheme to learn robust player policies. We further prove the framework can be extended to handle uncertainty about the bounded rationality of other players. In experiments, our framework learns robust policies in repeated N-player matrix games and, when applied to deep multi-agent reinforcement learning, can scale to complex spatiotemporal games. For example, it learns robust AI tax policies that improve welfare by up to 15%, even when taxpayers are boundedly rational.
BibTeX
@misc{zhao2021ermas,
  author = {Eric Zhao and Alexander R. Trott and Caiming Xiong and Stephan Zheng},
  title = {{ERMAS: Becoming Robust to Reward Function Sim-to-Real Gaps in Multi-Agent Simulations}},
  year = {2021},
  eprint = {2106.05492},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2106.05492},
}
Figure