RMIO: A Model-Based MARL Framework for Scenarios with Observation Loss in Some Agents

Zifeng Shi, Meiqin Liu, Senlin Zhang, Ronghao Zheng, Shanling DongarXiv 2024 · DOI · arXiv

Introduction

Most Multi-Agent Reinforcement Learning (MARL) research operates under the assumption of a Decentralized Partially Observable Markov Decision Process (Dec-POMDP), where every agent receives a reliable, albeit limited, local observation at every time step. However, real-world autonomous systems frequently encounter sensor failures, network packet loss, and environmental interference. When an agent receives a null observation ($\emptyset$), standard policies typically fail because they are not trained to handle missing data.

RMIO (Robust, Model-based, Incomplete Observation) addresses this gap by providing a mechanism to reconstruct and correct missing information on the fly, treating the world model as a redundant sensor system that "hallucinates" missing inputs based on historical context and inter-agent communication.

Methodology

RMIO builds upon the Recurrent State Space Model (RSSM) architecture. It introduces a two-stage process to handle observation loss: Observation Reconstruction and Correction.

Observation Reconstruction and Correction

When an agent loses its observation, RMIO uses its internal "Prior Model" to predict the most likely latent state, followed by an "Observation Predictor" to reconstruct the missing data. However, because world model predictions can drift over time, RMIO employs a Transformer-based Correction Block.

The RMIO pipeline for observation completion and correction, showing how missing observations are reconstructed and then refined using data from other agents.

This block integrates the reconstructed observations of "blind" agents with the real observations of "seeing" agents. By sharing information, the system can "triangulate" the missing state of a blinded agent far more accurately than that agent could do in isolation.

Architecture of the Correction Block

The Correction Block utilizes a self-attention mechanism to fuse information across the multi-agent system. This allows the model to weigh the relevance of different agents' observations when refining a specific reconstructed input.

Detailed architecture of the RMIO Correction Block, featuring encoders, MLPs, and a Self-Attention Layer for inter-agent information fusion.

Training and Stability

To ensure high performance even in perfect conditions, RMIO incorporates several refinements:

  • Hybrid CTDE Paradigm: While following Centralized Training, Decentralized Execution, it allows for limited, one-step communication during observation loss to synchronize historical states.
  • Reward Smoothing: A Gaussian kernel is used to handle noisy rewards.
  • Double Experience Replay Buffer: The framework mixes real and pseudo-trajectories to prevent the policy from overfitting to potential inaccuracies in the world model.

Evaluation and Results

The framework was evaluated on the StarCraft Multi-Agent Challenge (SMAC) and Multi-Agent MuJoCo (MaMuJoCo) benchmarks, modified to include a probability of observation loss ($p_{loss}$).

Key Findings

  • High Precision: In the 3s_vs_3z SMAC map, the Correction Block reduced the Mean Squared Error (MSE) of reconstructed observations from 0.0072 to 0.0009—nearly a tenfold improvement.
  • Robustness: RMIO remained stable even at high $p_{loss}$ levels where baselines like MAPPO and MAMBA collapsed. Remarkably, RMIO maintained functionality even when some agents suffered permanent observation loss ($p_{loss} = 1.0$).
  • State-of-the-Art Performance: In standard settings without loss, RMIO outperformed existing methods. For instance, in the 2s3z map, RMIO achieved a 98% win rate, significantly higher than MAMBA (78%) and MAG (81%).

Conclusion

RMIO demonstrates that "hallucination" through a world model, combined with inter-agent error correction, can effectively mitigate the impact of sensor failures in MARL. While the framework faces challenges in high-difficulty tasks with very large agent counts—where the joint observation space becomes increasingly complex—it represents a significant step toward deploying robust MARL systems in unpredictable real-world environments. Future work will explore extending this framework to adversarial scenarios and increasing its adaptability to extreme environmental complexity.

Summary updated

Abstract

In recent years, model-based reinforcement learning (MBRL) has emerged as a solution to address sample complexity in multi-agent reinforcement learning (MARL) by modeling agent-environment dynamics to improve sample efficiency. However, most MBRL methods assume complete and continuous observations from each agent during the inference stage, which can be overly idealistic in practical applications. A novel model-based MARL approach called RMIO is introduced to address this limitation, specifically designed for scenarios where observation is lost in some agent. RMIO leverages the world model to reconstruct missing observations, and further reduces reconstruction errors through inter-agent information integration to ensure stable multi-agent decision-making. Secondly, unlike CTCE methods such as MAMBA, RMIO adopts the CTDE paradigm in standard environment, and enabling limited communication only when agents lack observation data, thereby reducing reliance on communication. Additionally, RMIO improves asymptotic performance through strategies such as reward smoothing, a dual-layer experience replay buffer, and an RNN-augmented policy model, surpassing previous work. Our experiments conducted in both the SMAC and MaMuJoCo environments demonstrate that RMIO outperforms current state-of-the-art approaches in terms of asymptotic convergence performance and policy robustness, both in standard mission settings and in scenarios involving observation loss.
BibTeX
@misc{shi2024rmio,
  author = {Zifeng Shi and Meiqin Liu and Senlin Zhang and Ronghao Zheng and Shanling Dong},
  title = {{RMIO: A Model-Based MARL Framework for Scenarios with Observation Loss in Some Agents}},
  year = {2024},
  eprint = {2411.19639},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2411.19639},
}
Figure