Multi-Agent Reinforcement Learning via Adaptive Kalman Temporal Difference and Successor Representation

Mohammad Salimibeni, Arash Mohammadi, Parvin Malekzadeh, Konstantinos N. Plataniotis

TL;DR

This paper introduces novel Kalman Filter-based frameworks, MAK-TD and MAK-SR, for Multi-Agent Reinforcement Learning (MARL). These methods address sample inefficiency and parameter sensitivity issues common in DNN-based MARL by leveraging Kalman Filtering's uncertainty modeling and online learning capabilities, demonstrating superior performance in complex multi-agent environments.

What Problem Does This Paper Address?

Multi-Agent Reinforcement Learning (MARL) is crucial for scenarios involving multiple interacting agents, from robotics to game playing. However, traditional Reinforcement Learning (RL) algorithms, both model-based and model-free, struggle with MARL due to the dynamic and often non-stationary nature of the environment from each agent's perspective. While Deep Neural Network (DNN) approaches have shown promise, they often suffer from significant drawbacks: overfitting, high sensitivity to hyperparameter tuning, and poor sample efficiency, requiring vast amounts of data and computational resources. This paper aims to bridge this gap by proposing a more robust and efficient alternative that can handle the complexities of MARL, particularly in continuous action spaces, while explicitly modeling uncertainty.

Methods

The core contribution of this paper is the introduction of two novel frameworks:

  • Multi-Agent Adaptive Kalman Temporal Difference (MAK-TD): This framework leverages the principles of Kalman Filtering (KF) to address parameter uncertainty in MARL. It utilizes Kalman Temporal Difference (KTD) to model the value function estimation as a filtering problem, allowing for online second-order learning and uncertainty estimation. This approach is designed to be more robust to parameter variations and improve sample efficiency compared to DNN-based methods.
  • Multi-Agent Adaptive Kalman Successor Representation (MAK-SR): This is an extension of MAK-TD that incorporates Successor Representations (SR). SRs are useful for learning representations of future state occupancies, which can accelerate learning when reward functions change. In MAK-SR, the SR learning process is also framed as a filtering problem using KTD, enabling uncertainty estimation for the SR itself.

Both frameworks employ Radial Basis Function (RBF) estimators to encode continuous action spaces into feature vectors. For learning localized reward functions, they utilize Multiple Model Adaptive Estimation (MMAE) to handle unknown observation noise covariances and mapping functions. This combination allows the frameworks to handle high-dimensional, continuous action spaces common in MARL.

Datasets

The authors evaluate their proposed MAK-TD and MAK-SR frameworks using a multi-agent extension of the OpenAI Gym benchmark. This benchmark provides a two-dimensional world designed to simulate cooperative, competitive, and mixed interaction scenarios. The environments are described as having continuous observation spaces and discrete action spaces, with agents receiving observations and rewards at each step.

Key Insights and Ideas

  • Kalman Filtering for MARL Robustness: The paper's central idea is that the inherent uncertainty modeling and online learning capabilities of Kalman Filters are well-suited to address the challenges of MARL, particularly the non-stationarity and parameter sensitivity that plague DNN-based methods.
  • KTD for Uncertainty in Value Functions and SR: Explicitly modeling the uncertainty in value function (MAK-TD) and Successor Representation (MAK-SR) estimations using KTD is shown to be crucial for improving robustness and sample efficiency.
  • RBFs for Continuous Action Spaces: The use of RBFs to encode continuous action spaces into feature vectors is a practical approach for handling high-dimensional action spaces in MARL.
  • MMAE for Reward Function Learning: Employing MMAE to learn reward functions addresses the common issue of unknown observation noise and mapping functions in real-world MARL problems.
  • MAK-SR's Sample Efficiency: The MAK-SR framework, by incorporating SR, demonstrates superior sample efficiency, meaning it can achieve good performance with fewer training episodes compared to other methods.

Key Results

The experimental results, conducted on the OpenAI Gym MARL benchmark across cooperative, competitive, and mixed scenarios, indicate that the proposed MAK-TD and MAK-SR frameworks significantly outperform state-of-the-art counterparts, including DQN, DDPG, and MADDPG.

  • Superior Performance: The MAK-SR framework, in particular, shows remarkable results, achieving better performance in terms of cumulative reward and requiring fewer steps to converge across various environments.
  • Improved Sample Efficiency: The frameworks demonstrate exceptional performance with a limited number of training episodes, a key advantage over DNN-based methods that often require extensive training.
  • Stability: The use of RBFs and the Kalman filtering approach contribute to the inherent stability of the implemented frameworks, as evidenced by consistently low loss values after initial training.
  • MAK-TD as an Alternative: While MAK-SR is more sample-efficient, MAK-TD is presented as a viable alternative, offering similar performance when SR representations are not transferable between different optimal policies.

Limitations and Future Work

  • Limited to Finite Action Spaces: The current MAK-SR and MAK-TD frameworks are designed for systems with a finite number of actions.
  • Future Work: Infinite Action Spaces: A significant direction for future research is to extend these frameworks to applications with infinite action spaces, which are common in continuous control problems. This would involve addressing the challenges of handling continuous action spaces more directly.
  • Transferability of SR: The paper notes that learned SR representations are not always transferable between different optimal policies, which is why MAK-TD is also presented as a robust alternative. Future work could explore methods to improve SR transferability in MARL.

Summary updated

Abstract

Development of distributed Multi-Agent Reinforcement Learning (MARL) algorithms has attracted an increasing surge of interest lately. Generally speaking, conventional Model-Based (MB) or Model-Free (MF) RL algorithms are not directly applicable to the MARL problems due to utilization of a fixed reward model for learning the underlying value function. While Deep Neural Network (DNN)-based solutions perform well, they are still prone to overfitting, high sensitivity to parameter selection, and sample inefficiency. In this paper, an adaptive Kalman Filter (KF)-based framework is introduced as an efficient alternative to address the aforementioned problems by capitalizing on unique characteristics of KF such as uncertainty modeling and online second order learning. More specifically, the paper proposes the Multi-Agent Adaptive Kalman Temporal Difference (MAK-TD) framework and its Successor Representation-based variant, referred to as the MAK-SR. The proposed MAK-TD/SR frameworks consider the continuous nature of the action-space that is associated with high dimensional multi-agent environments and exploit Kalman Temporal Difference (KTD) to address the parameter uncertainty. The proposed MAK-TD/SR frameworks are evaluated via several experiments, which are implemented through the OpenAI Gym MARL benchmarks. In these experiments, different number of agents in cooperative, competitive, and mixed (cooperative-competitive) scenarios are utilized. The experimental results illustrate superior performance of the proposed MAK-TD/SR frameworks compared to their state-of-the-art counterparts.
BibTeX
@inproceedings{salimibeni2022multi,
  author = {Mohammad Salimibeni and Arash Mohammadi and Parvin Malekzadeh and Konstantinos N. Plataniotis},
  title = {{Multi-Agent Reinforcement Learning via Adaptive Kalman Temporal Difference and Successor Representation}},
  booktitle = {Sensors},
  year = {2022},
  url = {https://openreview.net/forum?id=u6vXZOaJCaS},
}
Figure