Multiagent Reinforcement Learning Based on Structural Coordination

Yixuan Li, Yi Huang, Junlan Feng, Chao Deng, Chunyu Liu, Vincent Chau, Wanyuan Wang

Abstract

With the rapid development of agent technology, multiagent systems (MASs) have become a key technology for solving complex collaborative tasks. The key challenges in MASs are which agents to coordinate and how to coordinate with these agents. Traditional multiagent reinforcement learning (MARL) methods try to distribute the global team payoffs to each agent by value decomposition, but cannot perfectly represent the relationship between the global value function and the local value function. In this paper, we make full use of the local interaction structure between agents to model the coordination relationship, and propose a structural coordination-based MARL method, which can train multiagents’ policies in a coordinated manner. By exploiting the explicit coordination graph (CG), we first propose a coordinated MARL method. The coordinated MARL models the coordination between agents as a distributed constraint optimization problem (DCOP) and employs the Shapley value technique for credit assignment. Furthermore, in the case that the CG is unknown, we propose a self-attention mechanism to dynamically construct an implicit CG, allowing agents to consider the status and action information of other relevant agents. Experimental results show that the proposed MARL based on structural coordination outperforms the state-of-the-art methods in various multiagent coordination applications.
BibTeX
@inproceedings{li2024multiagent,
  author = {Yixuan Li and Yi Huang and Junlan Feng and Chao Deng and Chunyu Liu and Vincent Chau and Wanyuan Wang},
  title = {{Multiagent Reinforcement Learning Based on Structural Coordination}},
  booktitle = {PDCAT},
  year = {2024},
  doi = {10.1007/978-981-96-4207-6\_31},
}

Loading related content...

Figure