Introduction
Standard game-theoretic models typically rely on Expected Utility (EU) maximization, where agents seek to optimize their average rewards. While this risk-neutral approach is mathematically convenient, it often fails in safety-critical multi-agent systems (MAS) like autonomous driving. In these environments, an agent must account for the variance in rewards caused by the uncertain or potentially erroneous actions of others—the "trembling hand" problem. A self-driving car, for instance, should not merely optimize for average travel time but must actively minimize the risk of catastrophic failures resulting from other drivers' mistakes.
Methodology: Risk-Averse Equilibrium (RAE)
To address these shortcomings, this work introduces the Risk-Averse Equilibrium (RAE). The core of this solution concept is a modified utility function that explicitly penalizes reward variance:
$$r = \mathbb{E}[U] - \gamma \text{Var}(U)$$
Here, $\gamma$ serves as a tunable hyperparameter representing the agent's degree of risk aversion. The authors prove the existence of RAE for any finite N-player game using Kakutani’s Fixed Point Theorem.
To compute these equilibria in complex environments, two solvers are proposed:
- Stochastic Fictitious Play (SFP): Suitable for small-scale discrete games.
- PSRO-RAE: An extension of the Policy-Space Response Oracles (PSRO) framework for large-scale problems. It utilizes an augmented Markov Decision Process (MDP) that incentivizes both high returns and low per-step variance.
Experimental Evaluation
The RAE framework was tested across three diverse scenarios:
- Matrix Coordination Games: Randomly generated games (100 and 500 actions) containing high-risk, high-reward traps.
- Stag-Hunt Grid-World: A classic trust dilemma where agents choose between a high-reward "Stag" (requiring coordination) or a low-risk "Plant."
- Autonomous Driving: A two-way traffic simulation where agents decide when to overtake slow-moving vehicles while avoiding oncoming traffic.
Results and Insights
The experiments revealed several key advantages of the RAE approach:
The Efficient Frontier: Similar to financial portfolio theory, RAE allows agents to navigate an "efficient frontier" of strategies. By adjusting $\gamma$, agents can find the strategy with the minimum possible variance for a given level of expected reward.
Enhanced Safety and Robustness: In the autonomous driving task, RAE achieved a 7x reduction in crashes compared to the PSRO-Nash baseline. By penalizing variance, RAE naturally avoids strategies that depend on perfect opponent execution, making it inherently robust to "trembles." In the Stag-Hunt game, RAE agents successfully identified the high variance of the "Stag" strategy and pivoted to the safer "Plant" option when coordination was uncertain, whereas standard Nash solvers frequently failed.
Conclusion and Future Directions
While RAE provides a robust framework for risk management in MAS, challenges remain. The risk-aversion parameter $\gamma$ currently requires manual tuning, and the model penalizes all variance equally, including "upside" variance. Future work may focus on asymmetric risk measures like Conditional Value at Risk (CVaR) to specifically target downside risks like collisions while allowing for beneficial reward fluctuations. Additionally, establishing universal convergence guarantees for SFP across all game classes remains a theoretical priority.
Summary updated