Multi-agent reinforcement learning (MARL) traditionally operates in a "closed-world" setting where tasks and team compositions remain fixed. However, real-world applications demand "open-world" adaptability, where agents can coordinate with varying teammates across diverse, unseen objectives. Offline meta-reinforcement learning offers a promising path forward by training agents on static, pre-collected datasets to generalize to new tasks. Yet, extending this to multi-agent systems introduces two major hurdles: decentralized task identification and role information scarcity.
The Challenges of Open-World MARL
- Decentralized Task Identification: In decentralized systems, agents typically only observe a fraction of their environment. Without global information, inferring the overarching team goal (the "task") from a limited local view is exceptionally difficult.
- Role Information Scarcity: Effective teamwork relies on specialized roles (e.g., an "attacker" vs. a "defender"). Offline datasets rarely include these labels. Without an understanding of these roles, agents often default to homogeneous, ineffective behaviors that fail to generalize to new scenarios.
The Solution: The $D^2TR$ Framework
To overcome these challenges, researchers introduce Decentralized and Disentangled Task–Role Representation Learning ($D^2TR$). This novel framework combines information-theoretic distillation with the semantic reasoning capabilities of Large Language Models (LLMs) to achieve robust team coordination. The architecture is built on four core pillars:
- Mutual Information Knowledge Distillation (MIKD): $D^2TR$ bridges the gap between global context and local execution. During training, a "teacher" encoder processes global trajectory data. This global task knowledge is then distilled into "student" encoders—which only access local agent data—via mutual information maximization. This allows local agents to accurately identify team-wide tasks during decentralized execution.
- LLM-Assisted Role Labeling: Because roles are implicit in offline data, $D^2TR$ leverages LLMs to analyze agent trajectories. The LLM generates semantic descriptions and assigns distinct role labels to behaviors found in the training data.
- Disentangled Representation Learning: A dedicated Role Encoder uses contrastive learning to recognize these behavioral patterns. This explicitly separates what the task is (team-centric goals) from how the agent should behave (individual-centric roles).
- Transformer-Based Policy: The final policy utilizes a Transformer architecture conditioned on both task and role encodings, allowing it to dynamically handle variable numbers of agents and adversaries.
Key Insights
The framework demonstrates that global information is not strictly necessary at execution time if it is effectively distilled into local representations during training. Furthermore, using LLMs as semantic labelers—rather than active game-players—provides a high-level "behavioral logic" that raw neural networks struggle to extract from standard state observations (like coordinates and health bars). By disentangling the task from the role, agents can transfer learned skills to entirely new tasks. For example, an agent that learned a "kiting" role in one scenario can seamlessly apply that behavior to a novel objective.
Experimental Results
$D^2TR$ was rigorously evaluated on Cooperative Navigation (CN Target), StarCraft Multi-Agent Challenge (SMAC), and the highly randomized SMACv2, which is specifically designed to test generalization.
- Superior Generalization: In Out-of-Distribution (OOD) tasks—scenarios the agents never encountered during training—$D^2TR$ consistently outperformed existing baselines. For instance, in the CN Target environment, it achieved a 0.39 win rate compared to 0.28 for the next best non-meta baseline.
- Robustness to LLM Choice: The framework's success relies on the conceptual extraction of semantic roles rather than the specific quirks of a single model; performance remained stable regardless of whether GPT or DeepSeek was used for labeling.
- Ablation Success: Removing either the MIKD module or the Role Encoder resulted in significant performance drops, confirming that both team-level task alignment and individual-level role information are critical for coordination.
Limitations and Future Directions
While highly effective, $D^2TR$ has limitations. Labeling thousands of trajectories via LLM APIs incurs computational costs and time, even with pre-clustering techniques. Additionally, the current model assumes an agent's role remains static throughout a trajectory, whereas complex environments may require dynamic role-switching (e.g., transitioning from offense to defense mid-game).
Future research will explore scaling to larger models and applying the framework to embodied AI tasks, such as physical humanoid robots coordinating to move objects in real-world spaces.
Summary updated