Committed Imagination: Escaping Bad Team Equilibria with a Learned Model

Teams of learning agents often settle on a safe but mediocre way of working together. The cause is well understood. When one agent tries the action that would be best if its partners also played their part, those partners are still exploring randomly. The trial looks bad, and the agent gives up. This failure is called relative over-generalization: each agent judges its options by averaging over its teammates' noise, so sharp high-reward joint strategies get washed out. It is worst when agents are physically different, such as a fast arm paired with a slow one or a drone paired with a ground vehicle. In those teams, "playing your part" means something different for each body, and nobody knows in advance how the bodies affect one another.

We propose to use a learned model of the environment to ask a question that real trial and error cannot answer cheaply: what would happen if every teammate committed to this joint plan, with no exploration noise? In imagination, teammates can be made to commit perfectly. The averaging that hides good joint strategies then disappears. Our method keeps a small archive of candidate team "modes", meaning shared plans that each agent interprets through its own body. It scores each mode by rolling out committed behavior in a learned per-agent world model, adds a bonus where the model is uncertain, and spends real environment interaction only on the modes that look best. A colleague could summarize it in two sentences: the learned model removes the teammate noise that traps teams in bad equilibria, and mode codes let physically different agents commit to the same plan without communicating at test time.

Core Claim

Each of the three directions has a method, but no method combines all three. Maximum Entropy Heterogeneous-Agent Reinforcement Learning (HASAC) handles heterogeneous agents and fights premature collapse through entropy. It is model-free, and its temperature is task-sensitive. Negotiated Reasoning (SVNR) provably removes relative over-generalization using particle-based policy updates and synchronized random seeds. It assumes identical action capabilities and also has no dynamics model. Empowering Multi-Robot Cooperation via Sequential World Models (SeqWM) learns a chain of per-agent world models and plans with sampling-based control. It does not represent model uncertainty or protect against settling on one interaction mode. R3DM does use a dynamics model to keep roles diverse, but that model is ego-centric and shared across homogeneous agents.

The claim is specific. Relative over-generalization is an estimation artifact: a joint strategy's value is measured under teammates' current noisy policies. A learned model lets us measure value under counterfactual commitment instead. Let $z$ be a discrete team-mode code, $\pi_i(a_i \mid o_i, z)$ agent $i$'s own policy network conditioned on that code, and $\hat{P}$ a learned dynamics model. The real-data estimate that misleads agents is

$$Q^{\text{noisy}}(z) = \mathbb{E}_{a_{-i} \sim \pi^{\text{explore}}_{-i}}\big[\textstyle\sum_t \gamma^t r_t\big],$$

where $a_{-i}$ are teammates' actions sampled with exploration noise and $\gamma$ is the discount. We instead score modes by

$$S(z) = \hat{V}^{\text{commit}}(z) + \beta\, \sigma(z),$$

where $\hat{V}^{\text{commit}}(z)$ is the return of imagined rollouts in which all agents play the noise-free mode of $\pi_i(\cdot \mid o_i, z)$, $\sigma(z)$ is the disagreement of a model ensemble on those rollouts, and $\beta$ trades optimism against trust. The gap between $Q^{\text{noisy}}$ and $\hat{V}^{\text{commit}}$ is exactly the quantity that model-free methods cannot see. The method succeeds or fails on whether exploiting this gap pays off despite model error.

Research Program

World model. We build one dynamics module per agent, each with its own action dimension and encoder, chained autoregressively as in SeqWM. Agent $i$'s next latent state is predicted from its own state and action plus the predicted latents of agents $1..i-1$. This follows SeqWM's argument that coupling must be modeled for physically interacting robots. We train an ensemble of five such chains. Disagreement across the ensemble supplies $\sigma(z)$, the epistemic uncertainty that SeqWM lacks.

Execution without leakage. Learning Disentangled Multi-Agent World Model for Decentralized Control (DMAWM) shows that peer-conditioned latents leak teammate information and break decentralized execution. SeqWM's autoregressive chain seems to conflict with this. We resolve the conflict by role separation. The coupled model is used only during training, to score modes and generate imagined data. Policies see only their own observation and the mode code $z$. A probe test, adapted from DMAWM's teammate-action probe, will check that policy features do not encode unobserved teammate actions.

Mode archive and training loop. We maintain a small archive of $K$ mode codes, with $K$ around 4 to 8. This idea comes from S2Q in Retaining Suboptimal Actions to Follow Shifting Optima, which keeps runner-up joint actions alive. S2Q is limited to discrete value mixing with shared parameters. Our archive is continuous and model-scored, and each agent decodes a mode through its own body.

Each training episode proceeds in three steps. First, all agents draw the same $z$ from a softmax over $S(z)$ using a synchronized random seed, the decentralization trick borrowed from SVNR. Second, policies are trained with a sequential heterogeneous update in the style of HASAC or HAPPO (Trust Region Policy Optimisation in Multi-Agent Reinforcement Learning, Heterogeneous-Agent Reinforcement Learning), using real data plus short imagined rollouts. Third, an extra objective pushes modes apart in imagined outcome space, so the archive does not collapse to one mode. The weakest-scoring mode is periodically re-initialized. MF-PBT (Multiple-Frequencies Population-Based Training) is the cautionary reference here: greedy replacement collapsed diversity. Following its fix, we re-initialize slowly, at a fixed low rate. At test time agents play $\arg\max_z \hat{V}^{\text{commit}}(z)$, a constant, so no communication is needed.

Analysis. Using a two-agent differential game with known structure, we will characterize the conditions under which committed scoring identifies the global mode. The condition should depend on the model's error on the high-reward basin relative to the reward gap between basins. This tells us when $\beta$ must be large.

Baselines

The naive reference is HAPPO (Trust Region Policy Optimisation in Multi-Agent Reinforcement Learning): sequential heterogeneous trust-region updates with no diversity mechanism. That paper's own analysis notes it can sit in suboptimal Nash equilibria. Beating it only shows that diversity helps at all.

The strongest heterogeneous competitor is HASAC (Maximum Entropy Heterogeneous-Agent Reinforcement Learning). It reports the best result in 31 of 35 tasks across its benchmarks, and useful Bi-DexHands policies within 10 million steps. The file gives no per-task score I can quote. We will run its released configuration. We will also give it a temperature sweep with the same number of tuning runs our method gets for $\beta$ and $K$, because its weakness is temperature sensitivity and an untuned comparison would be unfair.

The strongest competitor on the over-generalization question itself is SVNR (Negotiated Reasoning). It reports about 10 (the global optimum) on Max of Three where baselines stay near 0, 4.76 on Particle Gather where MADDPG and PR2 get zero or negative return, and 8853 on HalfCheetah-2x3 versus 8662 for PR2 and 112 for MADDPG. On homogeneous tasks we compare against these reported numbers and also rerun SVNR. On heterogeneous variants we run SVNR with zero-padded action spaces, its most natural extension.

The model-based competitor is SeqWM (Empowering Multi-Robot Cooperation via Sequential World Models) with its sampling-based planner. For Gate, Shepherd, and Bi-DexHands, no reported result was found on Lacuna for SeqWM's exact returns, so we rerun it. All methods get the same environment-step budget, the same seeds (ten per task), and equal wall-clock for imagination. Model-based methods must not win by silently using more compute.

Two ablations isolate the key ingredient. The first, "noisy imagination", keeps everything identical except that imagined teammates act with their exploration noise. If this matches the full method, commitment is not the reason it works. The second, "optimism only", removes the mode archive and commitment and keeps only the ensemble bonus on a single policy. It tests whether plain model-based optimistic exploration is enough.

Beating the field means three things. On tasks with a known global mode, the full method must reach that mode in clearly more seeds than HASAC-tuned and SVNR. It must match or exceed their final return. It must reach SVNR-level return in fewer environment steps.

Why not just…?

Why not just tune HASAC's temperature harder? Entropy spreads each agent's own actions. Relative over-generalization comes from teammates' spread, so more entropy can make it worse. A well-tuned HASAC may still escape on easy landscapes, and this is a real threat on dense-reward MAMuJoCo tasks. The tuned-HASAC baseline settles it: if HASAC with equal tuning budget reaches the global mode as often as we do on heterogeneous Max of Three and Particle Gather, the proposal's central claim fails.

Why not just extend SVNR to padded heterogeneous action spaces? SVNR already removes over-generalization, so this is the most serious alternative. Two things count against it. Zero-padding is the shared-representation trap that Trust Region Policy Optimisation in Multi-Agent Reinforcement Learning shows can be exponentially costly in asymmetric games. SVNR is also model-free and its particle updates cost quadratic time in particle count, so under unknown dynamics it pays for every probe with real samples. If padded SVNR matches us on final return, our remaining claim is sample efficiency only, and we will report it that way.

Why not just add an entropy bonus or a higher temperature to SeqWM's planner? Sampling-based planning explores around the current plan. It reweights nearby trajectories and does not hold distant joint modes alive. It also has no notion of model uncertainty, so it cannot tell an untried basin from a bad one. The SeqWM-plus-temperature variant is included as a baseline. The "optimism only" ablation additionally tests whether uncertainty without commitment is enough.

Why not just run population-based training over teams? MF-PBT shows that standard PBT collapses diversity and underperforms random search: an IQM of 16,171 versus 17,713 on Brax Humanoid at one billion steps, against 23,793 for MF-PBT. A population of whole heterogeneous teams also multiplies sample cost by the population size under unknown dynamics. Our archive is effectively a population evaluated mostly in imagination, which is the point.

Proposed Experiments

Benchmarks fall into three groups. We first rerun SVNR's tasks unchanged (Max of Three, Particle Gather, HalfCheetah-2x3) to confirm we do not lose on homogeneous problems. We then build heterogeneous versions in which the dynamics are unknown and must be learned:

  1. Max of Three embedded in point-mass dynamics. The two agents differ in mass, maximum force, and action dimension, and the payoff depends on positions reached, so reward requires learning dynamics.
  2. Particle Gather with agents of different speeds and sizes.
  3. MAMuJoCo partitions in which agents control limbs with different joint counts, including the 17-agent Humanoid where shared-parameter MAPPO fails (Heterogeneous-Agent Reinforcement Learning).

Finally, Bi-DexHands tasks test real physical coupling between two hands.

Metrics are final return, the fraction of seeds reaching the known global mode, environment steps to reach SVNR's reported level, and how many distinct imagined outcomes the mode archive covers. We also run DMAWM's teammate-action probe on policy features.

Three checks are decisive:

  1. Commitment versus noisy imagination. On heterogeneous Max of Three, the full method must reach the global mode in most seeds while the noisy-imagination ablation behaves like HAPPO. Otherwise the core mechanism is not doing the work.
  2. Calibration of the commitment estimate. $\hat{V}^{\text{commit}}(z)$ must rank modes in the same order as their true committed returns, measured by deploying each mode noise-free in the real environment. This tests whether model error undermines the idea.
  3. Probe accuracy. Teammate-action probe accuracy on policy features should stay near DMAWM's disentangled level, about 42%, rather than MAMBA's 85%. This would confirm that a coupled training-time model does not leak into decentralized execution.