When a team of different kinds of agents learns together, for example seven marines, two marauders and one medic in a StarCraft battle, the designer has to decide how much of the neural network the agent types share. Sharing everything makes the agents behave alike. Sharing nothing leaves the lone medic learning from a tenth of the team's experience. Current methods make this choice once, with a fixed rule or a learned mask, and they make it in the same way for every agent type. We propose to treat it as a statistical estimation problem. Each agent's policy weights are a shared base plus an agent-specific deviation. A shrinkage prior, meaning a prior that pulls parameters toward zero unless the data insist otherwise, sits on that deviation. The posterior then decides how far each agent type moves from the base. Agent types with plenty of experience are free to specialize. Rare agent types are pulled back toward the shared network until their own data justify a departure.
The central claim is that in heterogeneous teams the right amount of sharing depends on how much data each agent type has, and it also differs by layer. A hierarchical prior with inferred per-agent, per-layer shrinkage should find this amount automatically. Deterministic masking and fixed-strength regularization cannot. The payoff is concentrated where current methods are weakest: rare agent types, heavily imbalanced team compositions, and new compositions seen at test time. As a byproduct, the posterior spread over each deviation gives a per-agent measure of how well that agent's specialization is supported by data. We test whether that measure is useful rather than assume it.
Where this sits
Heterogeneous multi-agent reinforcement learning (MARL) has converged on two ingredients. The first is training without full sharing, using sequential per-agent updates. Heterogeneous-Agent Reinforcement Learning introduces HAPPO and HATD3, which update agents one at a time so that each adapts to its predecessors. Its authors show that fully shared MAPPO fails to learn at all on the 17-agent MAMuJoCo Humanoid. Maximum Entropy Heterogeneous-Agent Reinforcement Learning adds an entropy bonus on actions (HASAC) and reports the best result on 31 of 35 tasks. We reuse this sequential training machinery unchanged. Both papers use point-estimate weights and either share parameters fully or not at all.
The second ingredient is partial sharing. Kaleidoscope is the closest prior work. It writes each agent's weights as a shared vector times a learned binary mask, $\theta_i = \theta_0 \odot M_i$, and adds a diversity penalty and periodic resets. Our proposal departs from it in three concrete ways. First, sharing is continuous and additive rather than on/off. Second, the strength of sharing is a random variable inferred for each agent and layer rather than an outcome of mask optimization under a global diversity weight. Third, nothing in Kaleidoscope ties an agent's degree of specialization to how much evidence that agent has collected. The last difference is the one the experiments are built around.
The probabilistic side supplies the method. Feature Preserving Shrinkage on Bayesian Neural Networks via the R2D2 Prior gives a prior with an infinite spike at zero and heavy tails, together with a hybrid Gibbs/variational fitting scheme. In single-network image tasks it reports 92.85% OOD AUROC (CIFAR-10 versus Fashion-MNIST) against 71.02% for deep ensembles. We move that prior from individual weights to agent-specific deviations. Its spike-and-tail shape is what partial pooling needs: most deviations are exactly unnecessary, and a few matter a lot.
Two papers constrain the design. Improving Bayesian Neural Networks by Adversarial Sampling shows that random weight samples drawn during training drag down performance: the worst sample reaches 44.92% on CIFAR-100 against a 53.21% ensemble. In a team, each agent's weight noise also becomes nonstationarity for every other agent. For this reason we keep the shared base deterministic, place distributions only on the deviations, and act with posterior means at execution. Distributionally Robust Cooperative MARL with Value Factorization proves that robustifying agents independently breaks team-level greedy action recovery. We avoid per-agent adversarial weight perturbation for the same reason. We work in actor-critic training, where that result does not bind directly, but we check empirically that sampled deviations do not degrade coordination.
The other bridging papers in the file do not attempt this. Context-Aware Bayesian Network Actor-Critic is Bayesian only in the graph over actions, and BATPAL is Bayesian only in its belief over adversary type. Neither places distributions on weights or on how much agents share.
Core Claim
We write agent $i$'s actor weights in layer $l$ as a shared base plus a deviation, $\theta_{i,l} = \theta_{0,l} + \delta_{i,l}$. The deviation gets a hierarchical shrinkage prior:
$$\delta_{i,l,j} \sim \mathcal{N}\!\left(0,\ \omega^2\, \phi_{i,l}\, \lambda_{i,l,j}^2\right), \qquad \phi_{i,\cdot} \sim \text{Dirichlet}(a), \quad \lambda_{i,l,j} \sim \text{heavy-tailed}.$$
Here $j$ indexes individual weights, $\omega^2$ is a global budget for how much the team deviates from sharing, $\phi_{i,l}$ splits agent $i$'s share of that budget across layers, and $\lambda_{i,l,j}$ lets individual weights escape shrinkage. This is the R2D2 decomposition applied to deviations rather than to raw weights. The posterior over $\delta_i$ is shaped by agent $i$'s own policy-gradient signal. That signal is weaker and noisier for agent types that appear rarely, so standard partial-pooling logic predicts that their inferred $\phi_{i,l}$ collapses toward zero and they inherit the shared network.
Two predictions follow and can be falsified. The first is that the learned deviation magnitude $\lVert \delta_{i,l} \rVert$ should increase with the amount of experience agent type $i$ has had, while a fixed-strength regularizer shows no such dependence. The second is that the advantage over fixed-strength sharing should grow as team composition becomes more imbalanced.
Research Program
We build the hierarchical actor, and optionally the critic, on top of the HAPPO and HASAC trainers. The deviations are fit by mean-field variational inference with reparameterized samples during rollouts. The shrinkage hyperparameters are updated with the Gibbs steps from the R2D2 paper. Memory overhead is two tensors (mean and scale) per agent-specific deviation, plus one shared base. That is below no-sharing and far below ensembles.
Where agents are all distinct, as in MAMuJoCo limbs, deviations are per agent. Where agents come in types, as in SMAC, deviations are tied within each type. That tying is exactly what makes the per-type data count differ.
The KL term on deviations is weighted by the number of on-policy transitions that agent type contributed. This weighting is what turns imbalance into different amounts of shrinkage. Execution uses the posterior-mean weights, so decentralized execution cost is identical to a deterministic network.
The analysis has three parts. The first measures deviation magnitude against per-type experience and against layer depth. The hypothesis is that early perception layers stay shared and late action layers specialize. The second is an imbalance sweep in which we vary how many units of the rare type appear. The third tests a new composition: we train on some SMACv2 compositions and test on compositions where a type is scarcer or appears for the first time. There, the rare type's posterior should sit near the shared base and degrade gracefully.
A secondary question is whether the posterior spread of a type's deviation predicts that type's value-estimation error on held-out states. We report this as a finding, not as a headline.
Baselines
The naive reference points are full parameter sharing with a one-hot agent ID, run as MAPPO with the tuning recipe of The Surprising Effectiveness of PPO in Cooperative Multi-Agent Games, and no sharing, run as HAPPO from Heterogeneous-Agent Reinforcement Learning. These bracket the sharing spectrum. For MAPPO the file records that it matched or beat prior methods on 10 of 14 SMAC maps. The file gives no win-rate number for either baseline on our maps.
The strongest existing method is Kaleidoscope. It is the only published approach that learns partial sharing for heterogeneous teams and is reported to beat both extremes on SMACv2 and MAMuJoCo. No reported result found on Lacuna for Kaleidoscope's exact SMACv2 win rates or MAMuJoCo returns, so we rerun it from its released configuration inside the same HAPPO-based trainer. Beating it means higher median win rate or return across seeds on the imbalanced scenarios, with non-overlapping interquartile ranges under marl-eval. It also means no loss on balanced scenarios.
The ablation that isolates our key ingredient is the MAP version of the same model: $\theta_0 + \delta_i$ with a single L2 penalty on $\delta_i$ whose strength is tuned by grid search. We also run a per-agent learned-scale point estimate, which is empirical Bayes without posterior uncertainty. If the full model does not beat these, the benefit comes from additive sharing and not from inference.
We include an ensemble-critic baseline, like Kaleidoscope's masked critics, for the uncertainty question. All methods get the same environment steps, network widths matched by active parameter count, the same number of seeds, and the same hyperparameter search budget in GPU hours. The Bayesian model's extra hyperparameters come out of that budget.
Why not just…?
Why not just add an L2 penalty pulling each agent toward the shared weights? This is the most serious alternative, because it is the MAP estimate of a Gaussian version of our model. Our argument against it is specific. A single tuned strength cannot be simultaneously weak for the seven-marine type and strong for the lone medic, and hand-tuning per type does not scale. The imbalance sweep and the learned-scale point-estimate ablation settle this directly. If a tuned global L2 matches us across the sweep, we will conclude that the Bayesian machinery is unnecessary for performance. The paper then becomes a negative result about inference plus a positive result about additive sharing.
Why not just use Kaleidoscope, since it already learns how much to share? Kaleidoscope's masks are fit by gradient descent under a diversity penalty whose weight is global. Nothing in its objective makes a data-poor agent share more. It is also binary, whereas a rare agent may need a small continuous adjustment to the shared weights rather than a pruned subnetwork. The per-type experience analysis tests this: we plot mask overlap with other types against experience for Kaleidoscope, and deviation size against experience for our model. It is possible Kaleidoscope already shows the right trend implicitly. If so, our claim narrows to the benefit of new-composition transfer.
Why not just give fully shared networks a larger capacity and an agent-ID input? Evidence in Heterogeneous-Agent Reinforcement Learning shows that shared MAPPO fails to learn on the 17-agent Humanoid, and Kaleidoscope reports that full sharing underperforms partial sharing on SMACv2. We still run a width-matched shared baseline so that capacity is not a confound.
Why not just use deep ensembles for uncertainty? Deep ensembles multiply memory by the ensemble size for every agent. In the single-network setting of the R2D2 paper they scored 71.02% OOD AUROC against 92.85% for the shrinkage BNN. That setting is not multi-agent RL, so we only compare them on the secondary held-out value-error question, where they may well win.
Proposed Experiments
The main testbed is SMAC and SMACv2. We use MMM2 because its composition is naturally imbalanced, and SMACv2 because it lets us control composition directly. We sweep the fraction of the rarest unit type from balanced down to a single unit and hold out compositions for testing.
MAMuJoCo covers continuous control with all-distinct agents. There we expect parity with no-sharing at lower memory rather than large gains, which is a useful check that the method does no harm. Level-Based Foraging, with agents at mixed skill levels, gives a cheap imbalance testbed for quick iteration. The metrics are test win rate or episodic return aggregated with marl-eval in BenchMARL, deviation-magnitude-versus-experience curves, and sample efficiency for the rare type measured as that type's own contribution to return where the environment allows attribution.
The decisive check is the imbalance sweep against the tuned-L2 and learned-scale ablations. The core claim holds only if the full model's margin over them grows as the rare type becomes scarcer, and if its learned deviation for that type shrinks correspondingly. A flat margin falsifies it. A second check is a coordination sanity test: sampled versus mean deviations during training, on the matrix coordination game and MMM2. This verifies that per-agent weight noise does not reproduce the coordination failure that the DrIGM paper proves for independent per-agent robustification. If sampling hurts, we fall back to mean-only rollouts with posterior updates applied only to the hyperparameters, and report that as a constraint.
References
- Heterogeneous-Agent Reinforcement Learning
- Maximum Entropy Heterogeneous-Agent Reinforcement Learning
- Kaleidoscope: Learnable Masks for Heterogeneous Multi-agent Reinforcement Learning
- Feature Preserving Shrinkage on Bayesian Neural Networks Via the R2D2 Prior
- Improving Bayesian Neural Networks by Adversarial Sampling
- Distributionally Robust Cooperative Multi-agent Reinforcement Learning with Value Factorization
- Context-Aware Bayesian Network Actor-Critic Methods for Cooperative Multi-Agent Reinforcement Learning
- Bayesian Robust Cooperative Multi-Agent Reinforcement Learning Against Unknown Adversaries
- The Surprising Effectiveness of PPO in Cooperative Multi-Agent Games