CODA: Coordination via On-Policy Diffusion for Multi-Agent Offline Reinforcement Learning

Marcel Hedman, Kale-ab Abebe Tessera, Juan Claude Formanek, Anya Sims, Riccardo Zamboni, Trevor McInroe, John Torr, Elliot FosongarXiv 2026 · arXiv

TL;DR

Offline multi-agent reinforcement learning often fails because agents cannot co-adapt when trained on static, pre-collected datasets. CODA resolves this by using a policy-conditioned diffusion model to generate synthetic trajectories that dynamically evolve alongside the agents' current policies, successfully restoring the crucial co-adaptation loop without requiring real-world environment interaction.

What Problem Does This Paper Address?

In online Multi-Agent Reinforcement Learning (MARL), agents learn by interacting with both the environment and each other. If Agent A changes its strategy, Agent B observes this change and adapts accordingly. This interactive feedback loop is essential for establishing coordination and strategy agreement.

However, in offline MARL, agents must learn exclusively from a fixed, pre-collected dataset. Standard algorithms treat this as a "Best Response Under Data" (BRUD) problem, optimizing each agent independently against stale teammate actions recorded in the dataset. Because the dataset is static, cross-agent adaptation cannot occur. Even if the dataset has perfect coverage of the state-action space, agents frequently converge to incompatible, uncoordinated joint behaviors because they cannot safely predict how their teammates are evolving during training.

Methods

The authors introduce CODA (Coordination via On-Policy Diffusion for Multi-Agent Reinforcement Learning), an algorithm-agnostic data augmentation module. Instead of generating a static augmented dataset before training begins, CODA dynamically generates synthetic trajectories during training.

CODA utilizes a centralized diffusion network architecture over full joint trajectories (states, actions, rewards). Crucially, it samples these trajectories using on-policy conditioning. As the agents' policies update during the training process, CODA shifts its generated trajectory distribution to track what the current joint policy would likely produce, while remaining safely anchored to the empirical data manifold.

The paper implements this conditioning via two distinct mechanisms:

  1. Classifier Guidance (Implicit Conditioning): Used when policies are complex neural networks (like in MaMuJoCo). It uses the gradients of the current joint policy log-likelihood to nudge generated actions toward regions consistent with the current decentralized policies.
  2. Classifier-Free Guidance (CFG / Explicit Conditioning): Used when a compact conditioning representation of the policy is available (like task embeddings or parameter vectors).

Datasets

The authors evaluate CODA on two main environments:

  • Continuous Polynomial Games: Differentiable, stateless toy environments (specifically the Multiplication Game and the Twin Peaks Game) designed to explicitly isolate and test coordination pathologies and strategy agreement.
  • MaMuJoCo (Multi-Agent MuJoCo): A popular, high-dimensional continuous-control benchmark. The authors test on 2HalfCheetah (using Replay, Good, Medium, and Poor dataset variants) and 4Ant-Replay.

Key Insights and Ideas

  • Coverage is Not Enough: A key takeaway is that simply increasing dataset coverage does not solve multi-agent miscoordination. If the training distribution doesn't change in response to policy updates, agents will still optimize against historical, uncoordinated actions.
  • Static Augmentation Fails: Traditional diffusion-based augmentation or reward-conditioned (Q-conditioned) generation still results in a static dataset. CODA's breakthrough is making data generation dynamic and tightly coupled with the training iteration.
  • Inherently Multi-Agent Guidance: CODA's guidance term optimizes the joint policy log-likelihood. This ensures that all agents' actions are nudged simultaneously toward mutually compatible strategies, capturing vital cross-agent correlations.

Key Results

  • Polynomial Games: In the Multiplication Game, standard MADDPG and static diffusion baselines converge to suboptimal behaviors determined by fixed dataset statistics. CODA successfully alters the effective training distribution, allowing agents to follow the correct joint gradient to the true optimum.
Figure 1. Multiplication game (R = axay). Reward takes values in [−1, 1]. Left: offline dataset used to train diffusion models and the baseline MADDPG policy. Middle: policy evolution during training. Right: returns during training (the final epoch equals the final test return for deterministic policies). Q-conditioned augmentation appears optimal only due to action-boundary effects (a ∈[−1, 1]2): the y-agent quickly saturates at the action bounds, so the deterministic gradient update acts solely through the x- agent, inflating returns (see Figure 2 where exploiting boundary- effects cannot reach an optimum). Sample efficiency plot contains no error bars as in this environment deterministic policies create deterministic rewards at test-time.
  • Twin Peaks Game: CODA successfully navigates the complex reward landscape to recover coordinated learning, whereas baseline methods collapse entirely to a suboptimal (0,0) state.
Figure 2. Twin Peaks game. Left: offline dataset used to train diffusion models and the baseline MADDPG policy. Middle: pol- icy evolution during training. Right: returns during training (the final epoch equals the final test return for deterministic policies). CODA’s on-policy conditioning again mitigates offline miscoor- dination. Sample efficiency plot contains no error bars as in this environment deterministic policies create deterministic rewards at test-time.
  • MaMuJoCo Benchmarks: CODA delivers strong quantitative gains on Replay and Good datasets, where expert trajectories are present. For example, on 2HalfCheetah-Replay, CODA achieves a mean score of 6925.02 compared to MADDPG+BC's 6854.97. The authors verify that increasing the guidance scale ($\lambda$) reliably steers generated trajectories toward high on-policy likelihood.
Figure 3. On-policy steering in MaMuJoCo (2HalfCheetah). Mean action log-likelihood of CODA-generated trajectories under the current policy πcurrent (Replay dataset). “Random” and “Expert” denote policies at initialization and after 300k training steps on the Replay dataset. Standard error across 8 seeds; horizon H = 20; batch size 2048 trajectories. Unguided diffusion corresponds to λ = 0.

Limitations and Future Work

CODA's main limitation is its dependence on the underlying dataset support. On "Poor" or "Medium" datasets, CODA does not significantly outperform unconditional diffusion. This happens because as the learned policy improves, the trajectories it wants to generate fall entirely outside the behavioral support of the poor dataset, limiting the effectiveness of policy-tilting without risking out-of-distribution errors.

For future work, the authors suggest exploring adaptive guidance scale selection and developing hybrid approaches that combine policy-conditioned diffusion with model-based rollouts (world models) to safely extrapolate and expand coverage beyond the static offline data.

Summary updated

Abstract

Offline multi-agent reinforcement learning (MARL) enables policy learning from fixed datasets, but is prone to coordination failure: agents trained on static, off-policy data converge to suboptimal joint behaviours because they cannot co-adapt as their policies change. We introduce CODA (Coordination via On-Policy Diffusion for Multi-Agent Reinforcement Learning), a diffusion-based multi-agent trajectory generator for data augmentation that samples conditioned on the current joint policy, producing synthetic experience which reflects the evolving behaviours of the agents, thereby providing a mechanism for co-adaptation. We find that previous diffusion-based augmentation approaches are insufficient for fostering multi-agent coordination because they produce static augmented datasets that do not evolve as the current joint policy changes during training; CODA resolves this by more closely simulating on-policy learning and is a meaningful step toward coordinated behaviours in the offline setting. CODA is algorithm-agnostic and can be layered onto both model-free and model-based offline reinforcement learning pipelines as an augmentation module. Empirically, CODA not only resolves canonical coordination pathologies in continuous polynomial games but also delivers strong results on the more complex MaMuJoCo continuous-control benchmarks.
BibTeX
@misc{hedman2026coda,
  author = {Marcel Hedman and Kale-ab Abebe Tessera and Juan Claude Formanek and Anya Sims and Riccardo Zamboni and Trevor McInroe and John Torr and Elliot Fosong},
  title = {{CODA: Coordination via On-Policy Diffusion for Multi-Agent Offline Reinforcement Learning}},
  year = {2026},
  eprint = {2604.23308},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2604.23308},
}

Loading related content...

Figure