TL;DR
Offline multi-agent reinforcement learning often fails because agents cannot co-adapt when trained on static, pre-collected datasets. CODA resolves this by using a policy-conditioned diffusion model to generate synthetic trajectories that dynamically evolve alongside the agents' current policies, successfully restoring the crucial co-adaptation loop without requiring real-world environment interaction.
What Problem Does This Paper Address?
In online Multi-Agent Reinforcement Learning (MARL), agents learn by interacting with both the environment and each other. If Agent A changes its strategy, Agent B observes this change and adapts accordingly. This interactive feedback loop is essential for establishing coordination and strategy agreement.
However, in offline MARL, agents must learn exclusively from a fixed, pre-collected dataset. Standard algorithms treat this as a "Best Response Under Data" (BRUD) problem, optimizing each agent independently against stale teammate actions recorded in the dataset. Because the dataset is static, cross-agent adaptation cannot occur. Even if the dataset has perfect coverage of the state-action space, agents frequently converge to incompatible, uncoordinated joint behaviors because they cannot safely predict how their teammates are evolving during training.
Methods
The authors introduce CODA (Coordination via On-Policy Diffusion for Multi-Agent Reinforcement Learning), an algorithm-agnostic data augmentation module. Instead of generating a static augmented dataset before training begins, CODA dynamically generates synthetic trajectories during training.
CODA utilizes a centralized diffusion network architecture over full joint trajectories (states, actions, rewards). Crucially, it samples these trajectories using on-policy conditioning. As the agents' policies update during the training process, CODA shifts its generated trajectory distribution to track what the current joint policy would likely produce, while remaining safely anchored to the empirical data manifold.
The paper implements this conditioning via two distinct mechanisms:
- Classifier Guidance (Implicit Conditioning): Used when policies are complex neural networks (like in MaMuJoCo). It uses the gradients of the current joint policy log-likelihood to nudge generated actions toward regions consistent with the current decentralized policies.
- Classifier-Free Guidance (CFG / Explicit Conditioning): Used when a compact conditioning representation of the policy is available (like task embeddings or parameter vectors).
Datasets
The authors evaluate CODA on two main environments:
- Continuous Polynomial Games: Differentiable, stateless toy environments (specifically the Multiplication Game and the Twin Peaks Game) designed to explicitly isolate and test coordination pathologies and strategy agreement.
- MaMuJoCo (Multi-Agent MuJoCo): A popular, high-dimensional continuous-control benchmark. The authors test on
2HalfCheetah(using Replay, Good, Medium, and Poor dataset variants) and4Ant-Replay.
Key Insights and Ideas
- Coverage is Not Enough: A key takeaway is that simply increasing dataset coverage does not solve multi-agent miscoordination. If the training distribution doesn't change in response to policy updates, agents will still optimize against historical, uncoordinated actions.
- Static Augmentation Fails: Traditional diffusion-based augmentation or reward-conditioned (Q-conditioned) generation still results in a static dataset. CODA's breakthrough is making data generation dynamic and tightly coupled with the training iteration.
- Inherently Multi-Agent Guidance: CODA's guidance term optimizes the joint policy log-likelihood. This ensures that all agents' actions are nudged simultaneously toward mutually compatible strategies, capturing vital cross-agent correlations.
Key Results
- Polynomial Games: In the Multiplication Game, standard MADDPG and static diffusion baselines converge to suboptimal behaviors determined by fixed dataset statistics. CODA successfully alters the effective training distribution, allowing agents to follow the correct joint gradient to the true optimum.
- Twin Peaks Game: CODA successfully navigates the complex reward landscape to recover coordinated learning, whereas baseline methods collapse entirely to a suboptimal (0,0) state.
- MaMuJoCo Benchmarks: CODA delivers strong quantitative gains on Replay and Good datasets, where expert trajectories are present. For example, on
2HalfCheetah-Replay, CODA achieves a mean score of6925.02compared to MADDPG+BC's6854.97. The authors verify that increasing the guidance scale ($\lambda$) reliably steers generated trajectories toward high on-policy likelihood.
Limitations and Future Work
CODA's main limitation is its dependence on the underlying dataset support. On "Poor" or "Medium" datasets, CODA does not significantly outperform unconditional diffusion. This happens because as the learned policy improves, the trajectories it wants to generate fall entirely outside the behavioral support of the poor dataset, limiting the effectiveness of policy-tilting without risking out-of-distribution errors.
For future work, the authors suggest exploring adaptive guidance scale selection and developing hybrid approaches that combine policy-conditioned diffusion with model-based rollouts (world models) to safely extrapolate and expand coverage beyond the static offline data.
Summary updated