When a team of learning agents faces a long stream of changing tasks, such as a hundred different kitchen layouts in the cooperative cooking game Overcooked, it loses more than task skills. It also loses its conventions, the unspoken agreements about who does what. The largest study of this setting reports deadlocks and "role collapse," where one agent idles while its partner does all the work. Even the best methods reach only about 0.30 normalized score on the hardest layouts (MEAL). Separately, work on single-agent RL for language models shows that policy-gradient training has a built-in drift toward low policy entropy (how spread out an agent's action choices are). Policies whose entropy has collapsed become rigid and can no longer be retrained on new tasks (Entropy-preserving reinforcement learning). This proposal tests one hypothesis that links the two results. In multi-agent continual learning, each agent's entropy collapse narrows the behavior its partners experience. The team therefore fits a brittle convention to a near-deterministic partner. When the layout changes, neither agent explores enough to renegotiate that convention. If the hypothesis holds, the collapse should compound with team size and could explain part of the "more agents" penalty.
The intervention is deliberately cheap. We do not add a diffusion model or a sampler. Each agent gets a feedback controller that holds its own policy entropy near a target throughout the task stream, applied through the policy's advantage term. This costs no extra forward passes and fits the fully GPU-vectorized training loop that makes 100-task experiments feasible. The contribution is primarily a causal diagnostic backed by a minimal fix: we intervene on entropy while holding everything else fixed, and measure whether coordination forgetting, forward transfer, and the agent-count penalty change.
Core Claim
We make three testable claims. First, per-agent entropy at the end of task $k$ predicts how fast the team learns task $k{+}1$ and how badly it forgets earlier tasks, beyond what the continual-learning regularizer explains. Second, directly clamping each agent's entropy near a target, with no task identity, no extra networks, and under 5% wall-clock overhead, reduces coordination forgetting on MEAL's hard Overcooked layouts. Third, the benefit grows with the number of agents, because collapse in one agent shrinks the exploration signal available to all the others.
Each claim can fail cleanly. If clamping entropy leaves forgetting and deadlock rates unchanged, the hypothesis is wrong, and coordination forgetting must come from somewhere else, such as representation drift in the shared trunk.
What Exists and What Is Missing
Three bodies of work motivate the proposal without doing it.
MEAL builds a JAX benchmark that runs 100+ cooperative tasks, 10 billion environment steps, in about four hours on one H100. It pairs standard continual-learning methods (EWC, Online EWC, MAS, A-GEM, PackNet) with multi-agent PPO variants (IPPO, MAPPO, HAPPO). It documents coordination forgetting and the more-agents penalty, but all of its policies are ordinary categorical or Gaussian policies with no entropy management.
Entropy-preserving reinforcement learning proves that policy gradients bias toward entropy reduction, because advantages correlate with log-probabilities. It proposes an advantage correction (REPO) and entropy-tracking asymmetric clipping (ADAPO), and it links preserved entropy to later retrainability. That work covers single-agent language-model tool use and math only. It reports 79% on AppWorld Test Normal and 71% on Test Challenge. It says nothing about teams, where one agent's entropy is part of another agent's environment.
Heterogeneous Agent Q-weighted Policy Optimization (HAQO) does bring entropy-regularized, multimodal diffusion policies to multi-agent RL. It reaches a 78.2% win rate on SMACv2 protoss_5_vs_5 against 65.2% for MAPPO. However, it is single-task, costs 1.5–2.5× per iteration, and scales linearly with agent count through sequential updates. That cost is incompatible with MEAL's throughput regime.
Trajectory Entropy Exploration maximizes a nearest-neighbor estimate of trajectory entropy in a learned embedding space to escape homogeneous behavior. It is also single-task and adds contrastive training plus neighbor search on every batch.
Other work constrains the design.
- Consolidation across multiple timescales in fast and slow successor features handles drift but forces plain SGD and runs at low throughput. We therefore avoid any mechanism that touches the optimizer.
- Sample Weight Decay shows that plasticity loss in single-agent RL has an identifiable, cheap-to-fix cause: gradient attenuation and rank collapse, countered by age-based replay weighting. That result is a template for the kind of single-cause diagnosis we attempt.
- Continual Learning as Computationally Constrained Reinforcement Learning frames continual learning as RL under a per-step compute budget. That framing is why we report every result against wall-clock and steps per second, not only per environment step.
Research Program
The mechanism is a per-agent entropy controller. For agent $i$ with policy $\pi_i(a \mid o)$ over local observation $o$, we replace the advantage $A$ used in the PPO loss with
$$\tilde{A}_i = A_i - \beta_i \big(\log \pi_i(a \mid o) - \bar{\ell}_i\big),$$
where $\bar{\ell}_i$ is the batch-mean log-probability for agent $i$ and $\beta_i \ge 0$ is a per-agent coefficient. This is the REPO-style log-likelihood correction, which directly offsets the covariance between advantage and log-probability that drives collapse. The coefficient is not a fixed hyperparameter. It follows a dual update toward a target entropy $\mathcal{H}^*$:
$$\beta_i \leftarrow \max\!\big(0,\ \beta_i + \eta\,(\mathcal{H}^* - \hat{\mathcal{H}}_i)\big),$$
where $\hat{\mathcal{H}}_i$ is agent $i$'s measured mean policy entropy on the current batch and $\eta$ is a small step size.
In expectation, a log-probability term in the advantage behaves much like an entropy bonus. We therefore do not claim the gradient term itself is new. What matters is that the coefficient is controlled per agent against a target, and that it runs continuously across task boundaries without a task ID. Per-agent control is the part aimed at role collapse. An idling agent's entropy falls to near zero on "stay," and its own $\beta_i$ rises while its partner's does not.
We will build this on MEAL's JAX IPPO, MAPPO and HAPPO implementations, stacked on top of Online EWC and PackNet. The design principle is to add the controller to the best existing continual method, not replace that method. The analysis side logs four quantities at every task boundary:
- per-agent entropy on held-out states;
- entropy restricted to states near chokepoints;
- the MEAL coordination signatures (deadlock episodes and per-agent idle fraction);
- learning speed on the next task.
We then fit, across seeds and methods, how well end-of-task entropy predicts next-task learning speed and forgetting, controlling for the continual method used.
Baselines
The naive reference point is MEAL's own recipe with a fixed default entropy coefficient: multi-head IPPO or MAPPO with Online EWC, and PackNet. MEAL reports that PackNet achieves top scores with near-zero forgetting and that HAPPO with Online EWC gives the best overall retention. The file reports roughly 0.30 normalized score on Level 3 layouts for the best methods and a forgetting score near 0.03 on Level 1 with standard EWC. No reported numeric average score for PackNet specifically was found on Lacuna. We will rerun these baselines in our pipeline rather than copy numbers, using identical layout sequences, seeds, step counts, and network sizes.
The strongest competitor on this specific question is a tuned static entropy bonus. It is the obvious way to keep entropy up, it is free, and the REPO paper's argument that static bonuses fail rests on language-model evidence, not small categorical team policies. For fairness, it gets exactly the same tuning budget as our controller. We sweep the bonus coefficient over the same number of configurations as we sweep $\mathcal{H}^*$ and $\eta$ combined, choose each by performance on a separate short validation sequence, and then run it on the full 100-task stream.
The second strong competitor is a single team-wide auto-tuned temperature, as in soft actor-critic, applied to PPO. It is the ablation that isolates per-agent control. If it matches our method, the per-agent part is unnecessary.
We also run Reset & Distill (Prevalence of Negative Transfer in Continual Reinforcement Learning). It resets the online learner at every task and distills into a retained actor. It reports near-100% success on hard Meta-World sequences where P&C and ClonEx fall below 40%, at about 3% extra training time. It is the natural competitor because resetting also restores high entropy, through a very different route.
Finally, we run a HAQO-style diffusion-policy MAPPO on a truncated sequence as an expressive-entropy reference, reported at matched wall-clock rather than matched steps.
Beating the baselines means the following. First, a statistically reliable gain over the tuned static bonus and the team-wide temperature on Level 3 average normalized score, with lower deadlock and idle fractions, at equal tuning budget and under 5% extra wall-clock. Second, higher forward transfer than Reset & Distill on the Easy→Medium→Hard curriculum, where MEAL reports that transfer nearly doubles hard-task performance compared with training on hard tasks alone.
Why not just…?
Why not just raise the entropy bonus? This is the most serious threat, and we treat it as a possible outcome, not a strawman. The theoretical reason it may fall short is that a fixed coefficient cannot respond to entropy that declines at different rates for different agents and tasks. The REPO paper documents a structural bias in policy gradients toward entropy loss, and in MEAL's role collapse one agent's entropy vanishes while its partner's may not. The equal-budget sweep described above settles the question. If a tuned static bonus closes most of the gap, we will report the diagnostic finding (entropy predicts and causally drives coordination forgetting) as the main result and the controller as a convenience.
Why not just reset the team at each task, as in Reset & Distill? Resetting removes negative transfer and restores entropy, but it also discards everything the team knows about coordinating. MEAL's curriculum result, where Easy→Medium→Hard training nearly doubles hard-layout performance, shows that forward transfer matters in cooperative settings. The curriculum comparison directly measures what resetting gives up. The distillation stage also inherits whatever convention the specialist settled on. We will test this by checking deadlock rates of the distilled generalist.
Why not just use a more expressive multimodal policy, as HAQO does? Diffusion policies address a different failure, averaging two valid conventions into an invalid one. That failure may also matter. But at 1.5–2.5× iteration cost with linear scaling in agents, they would turn MEAL's four-hour, 100-task run into something closer to a day. We compare at matched wall-clock on a shortened stream. If diffusion wins even there, this is a signal that multimodality, not entropy level, is the real bottleneck.
Why not just rely on Online EWC or PackNet? We do rely on them: the controller is added on top of both. Parameter isolation protects old weights, but it does nothing to make the policy explore on a new layout. The modularity results in Alleviating Catastrophic Forgetting with Modularity suggest that isolation also limits transfer over long streams. The test is whether the controller adds gains on top of each method or only on top of the weaker ones.
Proposed Experiments
The main testbed is MEAL's procedurally generated Overcooked, 100-task sequences mixing Easy, Medium and Hard layouts. We report average normalized score, forgetting, and forward transfer, following MEAL's definitions, plus MEAL's coordination signatures (deadlocks, per-agent idle fraction) and steps per second. To test the agent-count claim, we use MPE SimpleSpread and SMAX within MEAL at increasing team sizes. We also add a partner-change variant in which co-players are swapped mid-stream. MEAL leaves this partner non-stationarity unexamined, and entropy should matter most here.
Four checks are decisive:
- Causal test. We clamp entropy at several target levels, including an artificially low target, and see whether forgetting and deadlock rates move monotonically with the target. A low clamp that worsens coordination forgetting is as informative as a high clamp that improves it.
- Scaling test. The gain over the static-bonus baseline should grow with team size. A flat gain would contradict the compounding story.
- Precision cost. Within-task final score on Level 1 must not drop meaningfully, since Overcooked rewards precise timing and too much entropy could hurt it.
- Compute test. Steps per second must stay within 5% of the MEAL baseline.
As a secondary check, motivated by the precision finding in the REPO paper, we will confirm that the JAX pipeline's numeric precision is not itself contributing to entropy decay.
References
- MEAL: A Benchmark for Continual Multi-Agent Reinforcement Learning
- Entropy-preserving reinforcement learning
- Heterogeneous Agent Q-weighted Policy Optimization
- Toward Efficient Multi-Agent Exploration With Trajectory Entropy Maximization
- Balancing Plasticity and Stability with Fast and Slow Successor Features
- The Rank and Gradient Lost in Non-stationarity: Sample Weight Decay for Mitigating Plasticity Loss in Reinforcement Learning
- Continual Learning as Computationally Constrained Reinforcement Learning
- Prevalence of Negative Transfer in Continual Reinforcement Learning: Analyses and a Simple Baseline
- Alleviating Catastrophic Forgetting with Modularity for Continuously Learning Linked Open Data