Abstract
Adaptive multi-agent cooperation with especially unseen partners is becoming more challenging in multi-agent reinforcement learning (MARL) research, whereby conventional deep-learning-based algorithms suffer from the poor new-player-generalization problem, possibly caused by not considering theory-of-mind theory (ToM). Inspired by the ToM personality in cognitive psychology, where a human can easily resolve this problem by predicting others' intuitive personality first before complex actions, we propose a biologically-plausible algorithm named the mixture of personality (MoP) improved spiking actor network (SAN). The MoP module contains a determinantal point process to simulate the formation and integration of different personality types, and the SAN module contains spiking neurons for efficient reinforcement learning. The experimental results on the benchmark cooperative overcooked task showed that the proposed MoP-SAN algorithm could achieve higher performance for the paradigms with (learning) and without (generalization) unseen partners. Furthermore, ablation experiments highlighted the contribution of MoP in SAN learning, and some visualization analysis explained why the proposed algorithm is superior to some counterpart deep actor networks.
Introduction
In Multi-Agent Reinforcement Learning (MARL), agents often excel when interacting with familiar training partners but struggle significantly when paired with novel collaborators—a phenomenon known as the new-player generalization problem. Unlike artificial agents, humans navigate these situations with ease by employing "Theory of Mind" (ToM). We quickly infer a partner's intentions and "personality" from their initial actions to adjust our own behavior.
Most current AI models treat partners as static elements of the environment or overfit to the specific quirks of their training teammates. This research introduces MoP-SAN, a framework that bridges this gap by combining ToM-inspired personality modeling with neuromorphic engineering to foster robust zero-shot coordination.
Methods: The MoP-SAN Architecture
The proposed architecture consists of two primary components designed to enhance adaptability and efficiency.
1. Mixture of Personality (MoP) Module
The MoP module models a partner's behavior not as a single fixed label, but as a continuous mixture of $k$ "base personalities." - Determinantal Point Process (DPP): To ensure these base personalities are distinct and effectively cover the behavioral space, the authors utilize DPP constraints. This encourages the model to develop "orthogonal" strategies during training, such as aggressive versus passive roles. - Personality Estimator (PE): A Transformer-based context encoder analyzes the partner's historical trajectory to predict their current mixture of traits, allowing the ego agent to anticipate actions.
2. Spiking Actor Network (SAN)
The "ego" agent's decision-making core utilizes Leaky Integrate-and-Fire (LIF) neurons. Unlike standard Deep Neural Networks (DNNs), SANs are biologically inspired and communicate via discrete spikes. The temporal dynamics and non-differentiable nature of these spiking neurons provide inherent noise resistance, which is critical for coordinating with unpredictable, unseen partners.
Experimental Setup
The framework was evaluated using the Overcooked environment via the PantheonRL framework.
In this benchmark, two agents must coordinate to prepare and serve onion soup. This requires high-level division of labor and precise spatial coordination. Agents were initially trained in pairs (Self-Play) and subsequently tested in zero-shot coordination scenarios with "unseen" agents they had never encountered during training.
Key Results and Insights
The experimental results demonstrate the superiority of the MoP-SAN approach in both learning and generalization phases.
- Zero-Shot Performance: In generalization tests with unseen partners, MoP-SAN achieved a score of approximately 142, significantly outperforming the standard SAN baseline (~111) and the DNN baseline (~41).
- Overcoming Overfitting: Standard DNN-based PPO models achieved high scores during training but failed during generalization. They essentially "memorized" their partners, whereas MoP-SAN's personality vector representation allowed it to adapt to "shades" of behavior in new partners.
- The SAN Advantage: Analysis suggests that the neuronal dynamics of Spiking Actor Networks act as a regularizer. This prevents the agent from becoming overly sensitive to the specific timing or quirks of a training partner, enhancing robustness.
- Ablation Studies: Removing the DPP module led to a drop in performance and higher variance. This confirms that forcing the model to recognize diverse partner types during training is essential for real-world flexibility.
Conclusion
MoP-SAN represents a significant step toward more human-like AI cooperation. By integrating psychological principles of personality with the efficiency of spiking neural networks, the model achieves state-of-the-art zero-shot coordination. Future work will explore scaling this approach to larger multi-agent groups and improving coordination with "irrational" or sub-optimal partners, such as human beginners.
Summary updated