Co-Observation Alignment for Sample-Efficient Heterogeneous Multi-Agent RL

When a team of different agents learns together, each agent usually learns to perceive the world from scratch. A robot's hip controller and knee controller, or a medic unit and a marine unit in a strategy game, read differently shaped sensor vectors. The two common fixes both fall short. The first gives every agent its own network, so every agent pays the full sample cost of learning what its inputs mean. The second pads every input to the same length and shares one network, which mixes up meanings and, for agents that behave differently, can produce much worse joint policies. Yet these agents often look at the same things at the same moment. Neighbouring joints sense the same joint angle. Several allied units see the same enemy. Each such shared sighting is a labelled pair: two differently formatted descriptions of one object. This pairing comes free from the simulator during centralized training.

We propose to use these shared sightings to align each agent type's private input format into one shared representation of the world's objects, which we call entities. We train one shared perception module on the pooled experience of all agents, and keep each agent's decision-making head separate. The claim is that most of the extra sample cost of heterogeneous teams comes from learning perception many times over, not from learning behaviour. If so, aligned shared perception should cut the environment steps needed by strong heterogeneous-agent methods. It should do so without the exponential penalty that full parameter sharing carries. It should also let those methods use more gradient updates per environment step without collapsing.

Core Claim

In cooperative heterogeneous multi-agent reinforcement learning (MARL), agents' observations overlap. That overlap gives a supervised alignment signal, which is stronger than the unpaired geometric or distributional matching used in prior cross-domain work. Training type-specific input encoders to agree on co-observed entities makes a shared trunk trainable on all agents' data. The gain should grow with the amount of observation overlap. A shared encoder trained only by the reinforcement-learning loss, with no alignment objective, should not recover the gain. That second prediction is what separates the idea from "just share a trunk."

Position Relative to Prior Work

The proposal sits in the middle of an open dispute. Parameter Sharing For Heterogeneous Agents in Multi-Agent Reinforcement Learning makes the case for sharing. It zero-pads observations and tags them with agent identity so that one network can represent all heterogeneous agents. It argues that separate networks are prohibitively expensive. Heterogeneous-Agent Reinforcement Learning and Trust Region Policy Optimisation in Multi-Agent Reinforcement Learning make the case against sharing. They prove that parameter sharing can be exponentially suboptimal in the number of agents. They give each agent its own network and update agents one at a time, using a decomposition of the team's advantage into per-agent terms (HAPPO, HATD3).

We take each side's justified point. Behaviour stays per-agent and sequential, which respects the negative result. Perception is shared, which addresses the sample cost Terry et al. identify. Padding is replaced by learned alignment. Neither line of work aligns heterogeneous inputs.

The alignment machinery comes from heterogeneous graph learning. That field projects each node type with its own learned matrix into a common space, as in Heterogeneous Attributed Network Embedding with Graph Convolutional Networks and Universal Network Representation for Heterogeneous Information Networks. We reuse these per-type projections. Those works learn them from graph structure, not from paired views.

FUG learns a basis transform for graphs of arbitrary feature dimension with a contrastive loss that keeps representations spread out. We borrow its uniformity term as a guard against representation collapse. HGOT aligns views by optimal transport without pairs. That is appropriate when no correspondence exists, but it discards the exact pairs available here. We test optimal transport as an alternative rather than adopting it.

On the reinforcement-learning side, Cross-Domain Transfer in Reinforcement Learning using Target Apprentice aligns two single-agent tasks with an offline manifold alignment. It reports that static alignment alone was 40x worse than its adaptive version on bicycle balancing. That result motivates aligning online, during training, rather than once beforehand. PerSim and V2A pool data across heterogeneous embodiments. Both assume identical state and action dimensions, which is the assumption we drop.

V2A also constrains the design. It shows that pooled transitions cause value misassignment unless value estimates are conditioned on agent identity. For that reason our value and policy heads stay agent-specific.

Research Program

Each agent $i$ has a type $\tau_i$. A small type-specific encoder $f_{\tau}$ splits the agent's raw observation into per-entity feature blocks, using the layout the environment documents, and projects each block into a token in $\mathbb{R}^d$. A shared attention trunk then processes the agent's set of tokens. Processing a set this way, as in the entity-set encoder of HAFT, handles a variable number of visible entities. Per-agent actor heads and a centralized critic sit on top of the trunk.

During centralized training the simulator reports which entity each token refers to. Let $z_{i,e,t}$ be agent $i$'s token for entity $e$ at time $t$. Let $\mathcal{C}_t$ be the set of pairs $(i,j,e)$ in which two agents of different types both observe entity $e$. The alignment loss is a contrastive objective among the entities present at the same time step:

$$\mathcal{L}_{\text{align}} = -\sum_{(i,j,e)\in\mathcal{C}_t} \log \frac{\exp(\langle z_{i,e,t}, z_{j,e,t}\rangle/T)}{\sum_{e'} \exp(\langle z_{i,e,t}, z_{j,e',t}\rangle/T)}$$

Here $T$ is a temperature and $e'$ ranges over entities agent $j$ observes at time $t$. Restricting negatives to the same step keeps the cost linear in batch size. We add FUG-style uniformity. Execution needs no entity identifiers: once trained, $f_\tau$ is applied to the raw observation alone.

The trunk is trained by $\mathcal{L}_{\text{align}}$ and the critic loss on transitions pooled across all agents. Policy-gradient updates do not flow into the trunk; this cut-off is a stop-gradient. Each agent's sequential HAPPO or HATD3 update is therefore a per-agent update over a representation held fixed during that step. This keeps the structure of the sequential update scheme intact. We will check empirically that monotonic-improvement behaviour survives, rather than claim it formally.

We instantiate this on HATD3, the off-policy method HARL reports as its most sample-efficient on Multi-Agent MuJoCo (MAMuJoCo). Off-policy learning is where pooled replay and higher replay ratios matter. We also instantiate it on HAPPO for comparison. A third analysis tracks the dormant-neuron ratio as the replay ratio rises. Higher Replay Ratio Empowers Sample-Efficient Multi-Agent Reinforcement Learning found this ratio drives collapse at extreme replay ratios. We test whether a pooled, aligned trunk delays that collapse.

Baselines

All methods share the same data, network width budget, total parameter count within 10%, and environment-step budget. Each gets the same hyperparameter search budget of 20 configurations, run over 5 seeds.

The naive reference is zero-padding with agent-identity indication and one fully shared network, following Parameter Sharing For Heterogeneous Agents. It is the cheapest way to get sharing. Beating it confirms that alignment adds something over padding. It is expected to fall short on tasks where agents' behaviours conflict.

The strongest existing method is HATD3 (and HAPPO) with fully independent networks from Heterogeneous-Agent Reinforcement Learning. It is the reported leader on heterogeneous benchmarks, and it embodies the "independence is mandatory" position. We found no reported numeric MAMuJoCo return for HATD3 on Lacuna. The paper states only that HATD3 "achieves new state-of-the-art results for off-policy algorithms," and that MAPPO fails on 17-agent Humanoid. We will therefore rerun it from its released configuration. Beating it means matching its final return while needing at least 2x fewer environment steps to reach 90% of that return. It also means no drop in final return beyond seed noise.

Independent HATD3 also runs at replay ratios of 2 and 4. On the SMAC benchmark, higher replay ratios turned 0% win rates on Super Hard maps into 60–80% within 2M steps, and let QMIX at RR=4 reach 90% on 3s_vs_5z in about half the baseline time (Higher Replay Ratio). For the SMACv2 extension, the reference is E2GN2, reported to reach target win rates 2–5x faster than standard graph networks (Boosting Sample Efficiency … via Equivariance).

Three ablations isolate the key ingredient. In the shared-trunk-no-alignment model, per-type projections and the shared trunk are trained only by the critic loss. In the per-agent-auxiliary model, each agent gets a self-supervised loss (latent next-step prediction) with no cross-agent pairing. This controls for the possibility that any auxiliary loss would help. In the unpaired-OT model, the contrastive loss is replaced by an HGOT-style sliced-Wasserstein distance between type token distributions. This tests whether the pairs matter.

Why not just…?

Why not just share a trunk with per-type input layers and let the RL loss align it? This is the most serious threat. If it matches our method, the idea reduces to standard architecture plumbing. Reward gradients are sparse and noisy, so we expect an RL-only trunk to align co-observed entities slowly. The shared-trunk-no-alignment ablation settles this directly. We also measure cross-type token agreement for co-observed entities over training. If the ablation closes more than half the gap to our method, we will report the paired-alignment loss as a minor contribution.

Why not just keep independent networks and raise the replay ratio? The Higher Replay Ratio results show this is a cheap and large win. We treat it as complementary rather than competing. The test is whether our method at RR=2–4 beats independent HATD3 at its best replay ratio. The paper's own collapse at RR=32 gives room for a second test: whether a pooled trunk pushes the collapse point higher.

Why not just pad and share, as Terry et al. do? Their own evidence is mixed. Padding was necessary in Prospector. HARL proves exponential suboptimality for fully shared policies and calls padding unstable. Still, the padded baseline is run head-to-head. If it wins on low-overlap tasks, that is informative rather than embarrassing.

Why not just align with PCA or offline manifold alignment? FUG reports knowledge loss from PCA. Target Apprentice reports static alignment 40x worse than its adaptive variant. Both findings come from other settings, so we include a PCA-to-$d$ baseline fit on early rollouts and frozen thereafter.

Proposed Experiments and Decisive Checks

The main testbed is MAMuJoCo: HalfCheetah 6x1, Ant 4x2, Walker 2x3 and 17-agent Humanoid. Its observation-depth setting (obsk), which controls how many neighbouring joints each agent sees, gives a controlled overlap knob. We sweep obsk from 0 to 2. Results are steps-to-90%-of-final-return, area under the learning curve, final return, wall-clock time, and dormant-neuron ratio, all on 5 seeds. Secondary tests are SMACv2 Protoss and Terran, where allies see the same enemies, and the heterogeneous-role MPE tasks. Neither MPE nor MAMuJoCo has reported numbers on Lacuna, so all comparisons are rerun.

The first decisive check is dose-response. At obsk=0 there is little co-observation, and our method should collapse to roughly the shared-trunk-no-alignment ablation. The gain should rise with obsk. A flat gain across obsk would mean the benefit comes from something other than alignment.

The second check is that the advantage over independent HATD3 should widen with agent count, from 6 agents up to 17. Perception is the part learned redundantly, so more agents means more redundancy removed.

The third check is negative transfer. On tasks where agents' roles differ sharply, per-agent heads must keep final return within noise of independent HATD3. A drop would indicate that sharing perception still carries part of the sharing penalty HARL warns about.