Energy-based Surprise Minimization for Multi-Agent Value Factorization

Karush Suri, Xiao Qi Shi, Konstantinos N. Plataniotis, Yuri A. LawryshynarXiv 2020 · OpenReview

Abstract: Multi-Agent Reinforcement Learning (MARL) has demonstrated significant success in training decentralised policies in a centralised manner by making use of value factorization methods. However, addressing surprise across spurious states and approximation bias remain open problems for multi-agent settings. Towards this goal, we introduce the Energy-based MIXer (EMIX), an algorithm which minimizes surprise utilizing the energy across agents. Our contributions are threefold; (1) EMIX introduces a novel surprise minimization technique across multiple agents in the case of multi-agent partially-observable settings. (2) EMIX highlights a practical use of energy functions in MARL with theoretical guarantees and experiment validations of the energy operator. Lastly, (3) EMIX extends Maxmin Q-learning for addressing overestimation bias across agents in MARL. In a study of challenging StarCraft II micromanagement scenarios, EMIX demonstrates consistent stable performance for multiagent surprise minimization. Moreover, our ablation study highlights the necessity of the energy-based scheme and the need for elimination of overestimation bias in MARL.

Introduction

In Multi-Agent Reinforcement Learning (MARL), agents typically operate within a "centralized training, decentralized execution" (CTDE) framework. While value factorization methods like QMIX have advanced this field by decomposing global rewards into individual agent contributions, two critical challenges persist: surprise in partially observable environments and overestimation bias.

Surprise refers to sudden, spurious changes in an agent's observations that can lead to erratic behavior and inefficient learning. Simultaneously, value-based methods often suffer from overoptimistic estimates that aggregate across multiple agents, resulting in suboptimal joint policies. EMIX addresses these issues by framing surprise minimization as an energy-minimization problem and integrating a Maxmin strategy to stabilize value estimation.

The EMIX Framework

EMIX enhances the standard QMIX architecture by incorporating three primary components designed to handle environmental volatility and estimation errors:

  1. Energy-based Surprise Value Function ($V_{surp}$): Surprise is modeled as a function of states ($s$), actions ($u$), and the standard deviation of state transitions ($\sigma$). This is implemented via an Energy-Based Model (EBM) where high energy levels correspond to high surprise.
  2. Mellowmax Energy Operator: To aggregate surprise across agents, EMIX utilizes the Mellowmax operator. The authors provide theoretical proof that this operator forms a contraction, ensuring that the surprise minimization process converges reliably.
  3. Maxmin Q-learning: To mitigate overestimation bias, EMIX maintains multiple target networks ($m$ estimators). By taking the minimum predicted value across these estimators, the agents adopt a more cautious and stable learning trajectory.
The EMIX architecture integrates an energy-based surprise value function with a standard mixer network to penalize environmental surprise.

Experimental Evaluation

The algorithm was rigorously tested on the StarCraft II Multi-Agent Challenge (SMAC), covering 12 micromanagement scenarios of varying complexity. EMIX was compared against established baselines including QMIX, VDN, COMA, and IQL.

Key Results

EMIX demonstrated superior performance, outperforming the standard QMIX agent in 9 out of 12 scenarios. A notable success was observed in the so_many_baneling map, which involves 27 enemy units and high state-to-state deviations. In this high-surprise environment, EMIX maintained a ~95% success rate, whereas competing methods experienced significant performance degradation.

Success rates across 12 StarCraft II scenarios, highlighting EMIX's superior performance and stability compared to standard MARL baselines.

Furthermore, ablation studies confirmed that the "TwinQMIX" component (using two target networks) significantly improved performance over standard QMIX, validating the hypothesis that overestimation is a major bottleneck in multi-agent value factorization. The energy-based scheme also proved more sample-efficient and stable across different random seeds than model-free alternatives like SMiRL.

Conclusion

EMIX introduces a robust framework for MARL that bridges statistical physics and reinforcement learning through energy-based surprise minimization. By treating predictable states as "boring" and intrinsically motivating agents to seek them, EMIX provides a powerful regularizer for volatile environments. While the use of multiple target networks increases computational overhead, the gains in stability and performance—particularly in complex, partially observable tasks—make it a significant advancement for multi-agent coordination.

Summary updated

Abstract

Multi-Agent Reinforcement Learning (MARL) has demonstrated significant success in training decentralised policies in a centralised manner by making use of value factorization methods. However, addressing surprise across spurious states and approximation bias remain open problems for multi-agent settings. Towards this goal, we introduce the Energy-based MIXer (EMIX), an algorithm which minimizes surprise utilizing the energy across agents. Our contributions are threefold; (1) EMIX introduces a novel surprise minimization technique across multiple agents in the case of multi-agent partially-observable settings. (2) EMIX highlights a practical use of energy functions in MARL with theoretical guarantees and experiment validations of the energy operator. Lastly, (3) EMIX extends Maxmin Q-learning for addressing overestimation bias across agents in MARL. In a study of challenging StarCraft II micromanagement scenarios, EMIX demonstrates consistent stable performance for multiagent surprise minimization. Moreover, our ablation study highlights the necessity of the energy-based scheme and the need for elimination of overestimation bias in MARL. Our implementation of EMIX can be found at karush17.github.io/emix-web/.
BibTeX
@misc{suri2020energy,
  author = {Karush Suri and Xiao Qi Shi and Konstantinos N. Plataniotis and Yuri A. Lawryshyn},
  title = {{Energy-based Surprise Minimization for Multi-Agent Value Factorization}},
  year = {2020},
  eprint = {2009.09842},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2009.09842},
}
Figure