Decentralized and Disentangled Task–Role Representation Learning for Generalizable Offline Multi-Agent Meta Reinforcement Learning

Lei Yuan, Ruiqi Xue, Yang YuICML 2026 · OpenReview

Multi-agent reinforcement learning (MARL) traditionally operates in a "closed-world" setting where tasks and team compositions remain fixed. However, real-world applications demand "open-world" adaptability, where agents can coordinate with varying teammates across diverse, unseen objectives. Offline meta-reinforcement learning offers a promising path forward by training agents on static, pre-collected datasets to generalize to new tasks. Yet, extending this to multi-agent systems introduces two major hurdles: decentralized task identification and role information scarcity.

The Challenges of Open-World MARL

  1. Decentralized Task Identification: In decentralized systems, agents typically only observe a fraction of their environment. Without global information, inferring the overarching team goal (the "task") from a limited local view is exceptionally difficult.
  2. Role Information Scarcity: Effective teamwork relies on specialized roles (e.g., an "attacker" vs. a "defender"). Offline datasets rarely include these labels. Without an understanding of these roles, agents often default to homogeneous, ineffective behaviors that fail to generalize to new scenarios.

The Solution: The $D^2TR$ Framework

To overcome these challenges, researchers introduce Decentralized and Disentangled Task–Role Representation Learning ($D^2TR$). This novel framework combines information-theoretic distillation with the semantic reasoning capabilities of Large Language Models (LLMs) to achieve robust team coordination. The architecture is built on four core pillars:

  • Mutual Information Knowledge Distillation (MIKD): $D^2TR$ bridges the gap between global context and local execution. During training, a "teacher" encoder processes global trajectory data. This global task knowledge is then distilled into "student" encoders—which only access local agent data—via mutual information maximization. This allows local agents to accurately identify team-wide tasks during decentralized execution.
  • LLM-Assisted Role Labeling: Because roles are implicit in offline data, $D^2TR$ leverages LLMs to analyze agent trajectories. The LLM generates semantic descriptions and assigns distinct role labels to behaviors found in the training data.
  • Disentangled Representation Learning: A dedicated Role Encoder uses contrastive learning to recognize these behavioral patterns. This explicitly separates what the task is (team-centric goals) from how the agent should behave (individual-centric roles).
  • Transformer-Based Policy: The final policy utilizes a Transformer architecture conditioned on both task and role encodings, allowing it to dynamically handle variable numbers of agents and adversaries.

Key Insights

The framework demonstrates that global information is not strictly necessary at execution time if it is effectively distilled into local representations during training. Furthermore, using LLMs as semantic labelers—rather than active game-players—provides a high-level "behavioral logic" that raw neural networks struggle to extract from standard state observations (like coordinates and health bars). By disentangling the task from the role, agents can transfer learned skills to entirely new tasks. For example, an agent that learned a "kiting" role in one scenario can seamlessly apply that behavior to a novel objective.

Experimental Results

$D^2TR$ was rigorously evaluated on Cooperative Navigation (CN Target), StarCraft Multi-Agent Challenge (SMAC), and the highly randomized SMACv2, which is specifically designed to test generalization.

  • Superior Generalization: In Out-of-Distribution (OOD) tasks—scenarios the agents never encountered during training—$D^2TR$ consistently outperformed existing baselines. For instance, in the CN Target environment, it achieved a 0.39 win rate compared to 0.28 for the next best non-meta baseline.
  • Robustness to LLM Choice: The framework's success relies on the conceptual extraction of semantic roles rather than the specific quirks of a single model; performance remained stable regardless of whether GPT or DeepSeek was used for labeling.
  • Ablation Success: Removing either the MIKD module or the Role Encoder resulted in significant performance drops, confirming that both team-level task alignment and individual-level role information are critical for coordination.

Limitations and Future Directions

While highly effective, $D^2TR$ has limitations. Labeling thousands of trajectories via LLM APIs incurs computational costs and time, even with pre-clustering techniques. Additionally, the current model assumes an agent's role remains static throughout a trajectory, whereas complex environments may require dynamic role-switching (e.g., transitioning from offense to defense mid-game).

Future research will explore scaling to larger models and applying the framework to embodied AI tasks, such as physical humanoid robots coordinating to move objects in real-world spaces.

Summary updated

Abstract

Offline meta reinforcement learning (RL) enables agents to learn a unified policy from multi-task offline data to support generalization in out-of-distribution (OOD) tasks. Recent approaches in single-agent RL tackle this by learning an efficient task representation to distinguish between tasks, showing promising adaptation ability. However, when extended to multi-agent settings, these methods struggle with decentralized task identification due to limited global information, and suffer from inefficient knowledge transfer in the absence of role information. To address this, we propose D$^2$TR, a novel context-based meta RL framework with efficient decentralized and disentangled task-role identification. Specifically, D$^2$TR first introduces mutual information knowledge distillation to align decentralized task representations with centralized task representations inferred from global trajectories, enabling efficient decentralized team-centric information identification. Next, D$^2$TR leverages a large language model to assign semantic roles to trajectories in offline data, and achieves effective individual-centric information inference by learning decentralized role representations. Extensive experiments conducted on commonly used multi-agent environments, including CN, SMAC, and SMACv2, demonstrate that D$^2$TR exhibits strong generalization performance to unseen tasks, outperforming prior multi-agent multi-task and context-based meta RL baselines.
BibTeX
@inproceedings{yuan2026decentralized,
  author = {Lei Yuan and Ruiqi Xue and Yang Yu},
  title = {{Decentralized and Disentangled Task–Role Representation Learning for Generalizable Offline Multi-Agent Meta Reinforcement Learning}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=rv1vx7zf2U},
}
Figure