Multi-agent reinforcement learning has two ways to let a team of agents choose actions within one time step. They can choose independently, each agent sampling its own action. Or they can choose in sequence: each agent sees what its predecessors just picked and chooses accordingly. This second option is called sequential action conditioning, and it applies the chain rule of probability across agents. Sequential choice clearly fixes some failures. When two agents must both swerve left or both swerve right, independent agents average the two options and crash. But sequential choice costs one forward pass per agent at test time, and the field disagrees about when it matters and whether the order of agents matters. The evidence is mostly "wins on 34 of 43 datasets," a vote count that cannot separate a real effect from seed noise. Nor can it say what kind of task benefits.
We propose to answer the question with random-effects meta-analysis, the statistical tool medicine uses to pool trials that genuinely differ. Each benchmark task is treated as a "study," and the benefit of sequential conditioning on that task as its "effect size." The model then splits the variation in that benefit into seed-to-seed noise and true task-to-task differences. The second ingredient is a cheap statistic of the offline dataset, computed before any reinforcement learning is run, that measures how strongly the agents' logged actions depend on each other. We test whether this statistic explains the task-to-task differences. If it does, practitioners get two things. One is a calibrated prediction, with an interval, of whether a new task needs sequential conditioning. The other is a principled way to avoid paying the sequential latency cost where it buys nothing.
Core Claim
The disagreement in the literature is a heterogeneity question that has been treated as a yes/no question. Offline Multi-Agent Reinforcement Learning via Sequential Score Decomposition (OMSD) reports that any agent order works, and that capturing some sequential dependency is enough. Multi-Agent Reinforcement Learning is a Sequence Modeling Problem (the Multi-Agent Transformer, MAT) reports that order changes learning speed. Both can be true if the average effect of order is near zero but varies substantially across tasks.
Our claim has three parts. First, the per-task benefit of sequential conditioning has large between-task variance. Second, most of that variance is explained by a pre-training measure of how dependent the agents' logged actions are. Third, agent count, the obvious heuristic, explains much less. OMSD's own theory motivates the third prediction. It shows that matching each agent's behavior separately recovers a valid joint behavior with probability falling like $2^{1-n}$ in the number of agents $n$, but only when the data contains several incompatible joint modes. Agent count alone therefore should not predict the gain.
Model and Diagnostic
For task $i$ we fit matched variants that differ only in whether actions are conditioned in sequence. Let $\hat{\theta}_i$ be the standardized difference in normalized return (Hedges' $g$ across seeds) and $\sigma_i^2$ its within-task sampling variance from seeds. The model is
$$\hat{\theta}_i = \beta_0 + \beta_1 D_i + \beta_2 \log n_i + u_i + \epsilon_i,\qquad u_i \sim \mathcal{N}(0,\tau^2),\ \ \epsilon_i \sim \mathcal{N}(0,\sigma_i^2),$$
where $\beta_0$ is the baseline effect, $\beta_1$ and $\beta_2$ are the slopes on the diagnostic and on log agent count, $u_i$ is the task's true deviation, and $\tau^2$ is between-task heterogeneity. Each task is weighted by $1/(\sigma_i^2+\tau^2)$, so a few very low-noise tasks cannot dominate.
The diagnostic $D_i$ estimates how much the agents' actions in the dataset depend on one another given the state. We compute it as a held-out log-likelihood gap between two behavior-cloning models, one autoregressive and one a product of independent per-agent models:
$$D_i = \frac{1}{|\mathcal{H}_i|}\sum_{(s,a)\in\mathcal{H}_i}\Big[\log \hat{\mu}^{\text{AR}}(a\mid s) - \sum_{j=1}^{n_i}\log \hat{\mu}_j(a_j\mid s)\Big].$$
Here $\mathcal{H}_i$ is a held-out split of task $i$'s dataset, and $a=(a_1,\dots,a_{n_i})$ is the logged joint action. $D_i$ is a plug-in estimate of the conditional total correlation of the logged joint action. It costs two supervised fits, far less than offline RL. Whether it is worth anything is an empirical question the meta-regression settles.
The order question gets its own model. Within each task we train several orderings (fixed index, random per episode, and a spatial-distance order in the style of Correlated Policy Optimization in Multi-Agent Subteams). We estimate a task-level random effect of order and a pooled mean. If the pooled mean is near zero and $\tau^2_{\text{order}}$ is significantly positive, OMSD and MAT are reconciled. If $D_i$ also moderates that order effect, the reconciliation has a mechanism.
Relation to Prior Work
The algorithmic line supplies the treatment we study. MAT decodes actions autoregressively over a transformer and proves monotonic improvement for any order. Oryx combines a retention-based sequence model with autoregressive implicit-constraint Q-learning and scales to 50 agents. OMSD regularizes decentralized policies with sequential diffusion scores. Cooperative Policy Agreement distills an autoregressive teacher through shared agreement keys. Multi-Agent Guided Policy Optimization keeps a centralized autoregressive guider close to what decentralized students can imitate.
All of these report vote counts or averages over benchmark suites. None estimates how much the benefit varies across tasks or why. We do not propose a new policy class. We reuse Oryx's architecture as the shared backbone and switch conditioning on and off.
Two papers bring mixed effects into decision making and are the closest connecting work. RoME splits a bandit's treatment effect into population and per-user parts with graph shrinkage. Federated Offline Reinforcement Learning splits hospital responses into shared and site-specific parameters. Both model heterogeneity inside the learner, across users or sites. We instead model heterogeneity in the effect of an architectural choice across tasks, which neither addresses.
Heterogeneity in Multi-Agent Reinforcement Learning measures how different agents are from one another in order to share parameters. We measure how dependent their actions are in order to decide whether to condition. The two are complementary, and its distances could serve as an additional moderator.
The statistical machinery is standard random-effects pooling of the kind applied across clinical cohorts, for example Vitamin D supplementation for tuberculosis prevention and Risk factors of epidural hematoma in patients undergoing spinal surgery. We borrow the method, not the findings.
Research Program
We build one controlled codebase on the Oryx backbone, in JAX for SMAX speed, with four arms. The first conditions in sequence with a fixed order, the second with a random order, and the third with a distance-based order. The fourth samples each agent independently, with the same network, the same critic, and the same number of parameters (the masked decoder simply sees no predecessor actions).
The task pool is every offline dataset the file documents with public collection protocols: the 43 SMAC/SMAX datasets, 16 MAMuJoCo datasets, and 6 RWARE datasets used by Oryx, plus Connector at several agent counts, T-Maze, and CPA's XOR, Permutation, Bridge, Sensor and Aloha puzzles. That is roughly 75 tasks, enough to estimate $\tau^2$ reasonably. We use restricted maximum likelihood with a Hartung–Knapp-style small-sample correction, and check robustness with a heavy-tailed random-effects distribution, because the Gaussian assumption is a known weak point.
Ten seeds per arm per task gives about 3,000 offline runs. We fix the tuning budget per arm to the same small sweep on a held-out subset of tasks, then freeze it. We compute $D_i$ for every task before any RL run and preregister the predicted sign and rough size of $\beta_1$.
Baselines
The naive reference point is the field's current practice: vote counting. We report the fraction of tasks on which sequential conditioning wins, as in Oryx's "matched or surpassed the current state-of-the-art on 34 out of 43 SMAC datasets," alongside the unweighted mean of normalized returns. We compute these on exactly the same runs as our model, so differences come only from the analysis. Vote counting is the baseline to beat because it is what the cited papers use. Our method beats it if it produces conclusions vote counting cannot. One example is a significant $\tau^2$ with an interval. Another is a reconciled order effect. A third is a case where vote counting and the pooled estimate disagree in sign or confidence.
The strongest simple statistical competitor is an unweighted mean or interquartile mean of normalized scores across tasks, with bootstrap confidence intervals over seeds and tasks. It is run on the same runs. We also run fixed-effect inverse-variance pooling, which assumes $\tau^2=0$, as the ablation that isolates the random-effects ingredient. The random-effects model beats these if its prediction intervals for held-out benchmarks achieve nominal coverage while the bootstrap and fixed-effect intervals undercover.
The strongest existing algorithm is Oryx. The file reports it surpassing baselines on all 6 RWARE datasets, on 14 of 16 MAMuJoCo datasets, and on 34 of 43 SMAC datasets. On the coordination puzzles, the reference is CPA, which reaches a return of about 74 on Aloha versus about 49 for AlberDICE and 100% success on XOR and Permutation. Our always-sequential arm reproduces Oryx. The comparison that matters practically is a diagnostic-gated policy that conditions in sequence only when the predicted gain's lower prediction bound exceeds zero. It wins if it stays within each task's seed noise of always-sequential Oryx on returns while running independent sampling, and so a single parallel forward pass, on a substantial fraction of tasks.
For the moderator, the competing heuristic is agent count alone. $D_i$ beats it if it explains a clearly larger share of $\tau^2$ in nested meta-regressions.
Why not just…?
Why not just always condition in sequence, since it never seems to hurt? For return alone this may be nearly right, and our analysis could confirm it. A pooled effect that is positive with small $\tau^2$ is a legitimate outcome, and we would report it as such. But sequential conditioning costs $O(n)$ forward passes at deployment, a limitation both MAT and the subteam paper flag. The subteam paper also found full correlation failed to learn at 15 agents in Predator-Prey, so "never hurts" is not established. The gated arm tests directly whether the latency can be avoided without losing return.
Why not just report interquartile means with bootstrap intervals? For the single question "is it better on average," this is a real competitor and may give similar pooled numbers. We include it as a baseline and expect agreement on the mean. What it does not give is a separation of seed noise from true task variation. It also does not give a prediction interval for an unseen task, or a test of which task properties explain the variation. Held-out-benchmark coverage is the experiment that shows whether that extra structure is worth having.
Why not just run a careful ablation on a few tasks, like Oryx's T-Maze ablation? A single-task ablation shows that sequential conditioning can be necessary, and Oryx's T-Maze result shows exactly that. But the OMSD–MAT disagreement likely arises because different papers looked at different tasks. Only a pooled analysis across many tasks, with moderators, can say whether both observations reflect one underlying dependence on how multimodal the data is.
Why not just use agent count as the rule of thumb? OMSD's $2^{1-n}$ argument depends on agent count only when the data contains several joint modes. Fully expert, single-mode data with many agents should show little benefit. The nested meta-regression decides this: if $\log n$ explains as much heterogeneity as $D_i$, we will say the simpler rule suffices.
Decisive Checks
The study succeeds or fails on four results. The first is whether the heterogeneity test rejects $\tau^2=0$ for the sequential-versus-independent effect. If it does not, the answer is "uniform effect" and the diagnostic is moot. The second is whether the preregistered slope $\beta_1$ is positive and $D_i$ explains a substantial share of $\tau^2$ beyond $\log n$. The third is whether the order analysis shows a near-zero pooled mean with nonzero spread, which would reconcile OMSD and MAT, or a consistent order effect, which would side with MAT.
The fourth is a leave-one-benchmark-family-out test. We fit the model on SMAC and MAMuJoCo, then predict per-task gains on RWARE, Connector and the CPA puzzles. We check that the 90% prediction intervals cover about 90% of observed gains, and that the gated policy holds Oryx-level returns on those held-out families. A negative result on the third test is still informative. It would mean the dataset-level dependence statistic is not where the sequential-conditioning benefit comes from, and future work on ordering and graph discovery would have to look at the reward structure instead.
References
- Offline Multi-Agent Reinforcement Learning via Sequential Score Decomposition
- Multi-Agent Reinforcement Learning is a Sequence Modeling Problem
- Correlated Policy Optimization in Multi-Agent Subteams
- Oryx: a Scalable Sequence Model for Many-Agent Coordination in Offline MARL
- Cooperative Policy Agreement: Learning Diverse Policy for Offline MARL
- Multi-Agent Guided Policy Optimization
- RoME: A Robust Mixed-Effects Bandit Algorithm for Optimizing Mobile Health Interventions
- Federated Offline Reinforcement Learning
- Heterogeneity in Multi-Agent Reinforcement Learning
- Vitamin D supplementation for tuberculosis prevention: A meta-analysis
- Risk factors of epidural hematoma in patients undergoing spinal surgery: a meta-analysis of 29 cohort studies