Abstract
Multi-Agent Reinforcement Learning (MARL) typically follows the Centralized Training with Decentralized Execution (CTDE) paradigm. However, existing CTDE methods often provide identical global information to all agents, ignoring individual needs, and suffer from performance degradation during execution when global data is unavailable. We propose Personalized Training with Distilled Execution (PTDE), a framework that personalizes global insights for each agent during training and employs knowledge distillation to enable agents to "hallucinate" these insights from local observations during execution. Evaluated on StarCraft II and Google Research Football, PTDE achieves state-of-the-art performance, significantly outperforming standard decentralized algorithms and maintaining high performance retention in fully decentralized settings.
1. Introduction
In Multi-Agent Reinforcement Learning (MARL), agents must coordinate to achieve a common goal. While centralized training allows access to the global state, decentralized execution limits agents to local, partial observations. Current approaches often fail to account for the specific informational needs of different agents or rely on global data that is unavailable in real-world deployment. PTDE addresses these issues by introducing a personalization layer and a knowledge distillation stage to bridge the gap between centralized training and decentralized execution.
2. Methodology
2.1 Global Information Personalization (GIP)
The core innovation of PTDE is the Global Information Personalization (GIP) module. Instead of providing the raw global state to all agents, GIP uses an Agent-Hyper Network to filter the global state into a personalized vector ($z^t_i$) tailored to each agent's local context.
As shown in Figure 2, the Agent-Hyper Network takes local observations and generates weights for a personalization network, which then processes the global state $s_t$. This ensures that agents receive the most relevant information for their specific role and position.
2.2 Two-Stage Training and Distillation
PTDE employs a two-stage training process. In the first stage, agents are trained as "Teachers" using their personalized global vectors. This stage leverages the full power of centralized information to find optimal policies. In the second stage, a "Student" network is trained to predict the teacher's personalized vector ($z^t_i$) using only local observations. This allows the agent to "hallucinate" the necessary global insights during execution.
Figure 4 illustrates the integration of these components within a centralized mixing network, where local RNN features are combined with personalized global information to compute individual and total Q-values.
3. Experimental Evaluation
We evaluated PTDE on the StarCraft Multi-Agent Challenge (SMAC) and Google Research Football (GRF). These benchmarks require high levels of coordination and strategic planning. Additionally, we explored a novel application in Information Retrieval by treating documents as agents in a Learning to Rank (LTR) task using the MSLR-WEB30K dataset.
4. Key Results
PTDE demonstrates significant performance gains over standard CTDE baselines. In StarCraft II's "Super Hard" scenarios, PTDE achieved a 63.1% win rate, whereas standard QMIX and QPLEX failed to win any matches. Furthermore, the Performance Retention Ratio (PRR) analysis shows that agents can retain up to 90% of their "all-knowing" teacher's performance using only distilled local information during execution.
5. Conclusion
PTDE provides a robust framework for MARL by personalizing global information and effectively distilling it for decentralized use. By treating global knowledge as a latent feature to be inferred rather than a constant input, PTDE enables agents to maintain high performance in complex, real-world environments where global sensors are unavailable.
Summary updated