Introduction
Most Multi-Agent Reinforcement Learning (MARL) research operates under the assumption of a Decentralized Partially Observable Markov Decision Process (Dec-POMDP), where every agent receives a reliable, albeit limited, local observation at every time step. However, real-world autonomous systems frequently encounter sensor failures, network packet loss, and environmental interference. When an agent receives a null observation ($\emptyset$), standard policies typically fail because they are not trained to handle missing data.
RMIO (Robust, Model-based, Incomplete Observation) addresses this gap by providing a mechanism to reconstruct and correct missing information on the fly, treating the world model as a redundant sensor system that "hallucinates" missing inputs based on historical context and inter-agent communication.
Methodology
RMIO builds upon the Recurrent State Space Model (RSSM) architecture. It introduces a two-stage process to handle observation loss: Observation Reconstruction and Correction.
Observation Reconstruction and Correction
When an agent loses its observation, RMIO uses its internal "Prior Model" to predict the most likely latent state, followed by an "Observation Predictor" to reconstruct the missing data. However, because world model predictions can drift over time, RMIO employs a Transformer-based Correction Block.
This block integrates the reconstructed observations of "blind" agents with the real observations of "seeing" agents. By sharing information, the system can "triangulate" the missing state of a blinded agent far more accurately than that agent could do in isolation.
Architecture of the Correction Block
The Correction Block utilizes a self-attention mechanism to fuse information across the multi-agent system. This allows the model to weigh the relevance of different agents' observations when refining a specific reconstructed input.
Training and Stability
To ensure high performance even in perfect conditions, RMIO incorporates several refinements:
- Hybrid CTDE Paradigm: While following Centralized Training, Decentralized Execution, it allows for limited, one-step communication during observation loss to synchronize historical states.
- Reward Smoothing: A Gaussian kernel is used to handle noisy rewards.
- Double Experience Replay Buffer: The framework mixes real and pseudo-trajectories to prevent the policy from overfitting to potential inaccuracies in the world model.
Evaluation and Results
The framework was evaluated on the StarCraft Multi-Agent Challenge (SMAC) and Multi-Agent MuJoCo (MaMuJoCo) benchmarks, modified to include a probability of observation loss ($p_{loss}$).
Key Findings
- High Precision: In the
3s_vs_3zSMAC map, the Correction Block reduced the Mean Squared Error (MSE) of reconstructed observations from 0.0072 to 0.0009—nearly a tenfold improvement. - Robustness: RMIO remained stable even at high $p_{loss}$ levels where baselines like MAPPO and MAMBA collapsed. Remarkably, RMIO maintained functionality even when some agents suffered permanent observation loss ($p_{loss} = 1.0$).
- State-of-the-Art Performance: In standard settings without loss, RMIO outperformed existing methods. For instance, in the
2s3zmap, RMIO achieved a 98% win rate, significantly higher than MAMBA (78%) and MAG (81%).
Conclusion
RMIO demonstrates that "hallucination" through a world model, combined with inter-agent error correction, can effectively mitigate the impact of sensor failures in MARL. While the framework faces challenges in high-difficulty tasks with very large agent counts—where the joint observation space becomes increasingly complex—it represents a significant step toward deploying robust MARL systems in unpredictable real-world environments. Future work will explore extending this framework to adversarial scenarios and increasing its adaptability to extreme environmental complexity.
Summary updated