Agents that learn without hand-designed rewards often learn "skills": distinct, repeatable behaviours, each tied to a code $z$ that is fed into one shared policy network. The training signal comes from a second network, the discriminator, which tries to guess $z$ from the states the agent visits. The agent is rewarded when the guess is right. This objective asks every skill to behave differently from every other skill. Yet all skills are trained through the same weights, typically by concatenating $z$ onto the observation.
The multi-agent literature has already learned this lesson about agent identities. When agents with different roles share one network and differ only by an appended ID tag, the update that helps one agent damages another. This failure is called gradient interference, and adding capacity does not fix it (Scaling Multi-Agent Reinforcement Learning with Selective Parameter Sharing). We propose to test whether the same failure happens one level down, between skills inside a skill-conditioned policy. We also propose to test whether it is worse there, because the skill objective pushes skills apart by design.
The core claim has two parts. First, in multi-agent skill discovery the per-skill policy gradients conflict in direction, not just in scale. The skill objective rewards behavioural separation, so skills should disagree about which features to strengthen. This is the conflict type that the best current agent-level fix, HPS: Hyperspherical Parameter Sharing for Efficient Multi-Agent Reinforcement Learning, explicitly leaves unaddressed. Second, removing that conflict makes the discovered skills more reliable and cheaper to learn, and lets the high-level coordinator that sequences skills solve sparse-reward tasks with fewer environment steps. If skills turn out to conflict only in scale, as HPS found for agents, the cheap fix suffices. That would itself be a useful negative result.
Where this sits
Three multi-agent methods train a shared policy conditioned on skill codes, and all three treat the conditioning architecture as solved:
- Heterogeneous Skill Learning for Multi-agent Tasks (HSL) embeds discrete skills as continuous latents, rewards mutual information between skills and behaviour, and reassigns skills every $k$ steps. It runs everything through a standard shared MLP/GRU.
- Hierarchical Multi-Agent Skill Discovery (HMASD) uses a Transformer coordinator to assign team and individual skills, again with plain latent-conditioned executors. It reports that only about 24% of discovered skills contribute to winning.
- DGPO shows that one latent-conditioned network learns several winning strategies more than an order of magnitude faster than separate networks. This motivates sharing, but DGPO never asks what sharing costs.
These papers supply the host algorithms we modify.
The parameter-sharing papers supply the diagnostic tools and candidate fixes, but all of them operate on agent identity:
- SePS shows interference, not capacity, is the failure, and fixes it with static clusters.
- HyperMARL generates per-agent weights from a hypernetwork, at a parameter cost that grows with target network size.
- HPS decomposes gradients into a direction component and a scale component. It finds that agents agree on direction and disagree on scale, so it normalises shared weights onto a unit sphere and lets an agent-conditioned generator output scale and shift factors.
- SNP-PS carves fixed subnetworks per agent by pruning.
None of these has been applied to skill codes, which change within an episode and are optimised against a discriminator.
The closest prior combination is Celebrating Diversity in Shared Multi-Agent Reinforcement Learning (CDS). It pairs a mutual-information diversity reward with a shared-plus-independent value split. Its independent modules are per agent, not per skill, and HSL reports that CDS collapses on 27m_vs_30m from that overhead.
Research Program
Measurement. The first stage is a measurement study, and it determines the design. In HSL and HMASD executors trained with their published objectives, we compute per-skill policy gradients on minibatches stratified by active skill. We decompose each pairwise gradient comparison into the direction and scale components used by HPS, and track both over training. As a control, we run the same measurement on agent-ID-conditioned baselines, which reproduces HPS's finding for agents.
We then test whether high directional conflict between two skills predicts the outcomes that matter:
- the discriminator's accuracy on those skills dropping,
- the policy ignoring $z$ (measured as the action-distribution divergence across skills at matched states), and
- the coordinator stopping selecting those skills.
Intervention. The second stage builds a skill-decoupled executor in two increments.
- The minimal version follows HPS with $z$ in place of agent ID. Backbone weights are constrained to the unit sphere, and a small network maps the current skill code to per-layer scale and shift modulation.
- If stage one shows substantial directional conflict, the second increment adds low-rank, skill-indexed weight corrections to each backbone layer. For a weight matrix $W$, the effective weight under skill $z$ is $W + U_z V_z^\top$, where $U_z, V_z$ have rank $r \ll$ layer width and are produced from $z$ by a small generator. This lets skills rotate features, not just rescale them. Unlike a full hypernetwork, the added parameters grow only as $r$ times layer width, regardless of team size.
Gradients from a skill reach its own low-rank factors fully. They reach the shared backbone only through the normalised directions.
Baselines
Naive reference. The naive reference is the unmodified host algorithm, HMASD and HSL, with $z$ concatenated to the observation. It is run from the authors' code, with their hyperparameters and step budgets. On 0-1 sparse-reward SMAC, HMASD reports win rates between 40% and 100% depending on the scenario, where MAPPO and MAT score 0% (Hierarchical Multi-Agent Skill Discovery). On Overcooked it reports about 295 episode reward, against about 0.1 for exploration baselines. These are the numbers to beat. Beating them means either:
- a higher final sparse-SMAC win rate on the scenarios where HMASD is below 100%, or
- reaching HMASD's final win rate in measurably fewer steps across 5 seeds, with non-overlapping confidence intervals on steps-to-threshold.
For dense-reward Super-Hard SMAC, the strongest skill-based competitor is HSL. HSL reports roughly 10- and 20-point gains over CDS on 6h_vs_8z and 5m_vs_6m. No absolute HSL win rate is reported on Lacuna, so we rerun it ourselves.
Strongest parameter-sharing competitor. This is HPS applied to agent identity on top of the host method. Agents get sphere-normalised weights and agent-conditioned modulation, while skills stay concatenated. This isolates whether decoupling skills adds anything once agents are already decoupled.
Other competitors. We also run HyperMARL-style hypernetworks over skill codes, and per-skill independent networks.
Ablations. Two ablations isolate the key ingredient:
- skill-conditioned scale-and-shift modulation without sphere normalisation (plain conditioning), and
- sphere normalisation plus modulation without the low-rank directional term.
Fairness. All variants use the same environment-step budget and the same evaluation protocol. Each gets an equal hyperparameter search of 12 configurations. Parameter counts are reported against the HPS convention. HPS used 233% of base parameters on 27m_vs_30m, against over 2700% for no sharing.
Why not just…?
Why not just give each skill its own network? With a handful of skills this is affordable, and it removes interference completely, so it is a genuine threat. The counter-evidence is DGPO: shared latent-conditioned networks found all optimal Spread solutions 1.7 to 15 times faster than prior methods, and outperformed separate networks in sample efficiency. Our per-skill-network baseline settles the question directly. If it matches our method in steps-to-threshold at 4 to 8 skills, the honest conclusion is that sharing across skills is not worth preserving at that scale. We then still test whether the gap opens at 16 to 32 skills, where separate networks lose the shared physics representation.
Why not just make the shared network bigger? SePS shows that for agent IDs, added capacity under concatenation does not remove interference. We do not assume this transfers to skills. A width-matched concatenation baseline, with parameter count equal to our method, is included, and the gradient-conflict measurements show whether conflict falls with width.
Why not just apply gradient surgery, projecting away conflicting per-skill gradient components? This is a reasonable optimiser-level alternative that needs no architecture change. We include it as a baseline. We expect it to slow learning, because when conflict is pervasive the projection discards most of the signal. If it matches us, the architectural claim weakens to "conflict exists and must be handled somehow," and we will report it that way.
Why not just use HPS as published? Its own analysis states that it handles disagreement in scale but not direction. Stage one of our program measures directly whether skills disagree in direction. If they do not, HPS-with-skills is the answer and the low-rank term is unnecessary.
Proposed Experiments
Benchmarks. The main testbeds are 0-1 sparse-reward SMAC with HMASD as host, and Super-Hard SMAC (6h_vs_8z, 5m_vs_6m, corridor) plus 27m_vs_30m with HSL as host. These follow The StarCraft Multi-Agent Challenge. We add Overcooked, and the Google Research Football Academy scenarios 3_vs_1_with_keeper and Hard Counter-attack (Google Research Football). Hard Counter-attack matters specifically because HSL's skill selector struggles there under sparse reward.
Metrics. We report:
- win rate and steps-to-threshold,
- discriminator accuracy per skill,
- the cross-skill action divergence used to detect ignored codes,
- the fraction of skills whose ablation lowers win rate, following HMASD's relevance analysis, and
- the per-skill gradient direction and scale conflict.
Decisive checks. Three results would decide the claim:
- Mechanism. Directional conflict between skills must exceed that between agents in the same runs. If it does not, the core claim fails.
- Prediction. Within baseline runs, skill pairs with high directional conflict must show lower discriminator accuracy or lower coordinator usage. This links the measured conflict to outcomes rather than treating it as a curiosity.
- Benefit. Our executor must improve steps-to-threshold over both the host and HPS-on-agents, and the low-rank term must account for the gain beyond the scale-only ablation.
Untested hypothesis about skill relevance. We also test whether HMASD's 24% useful-skill fraction rises. We hold this prediction loosely, since skill relevance may be set by the task-agnostic objective rather than by interference. A null result there, alongside faster learning, would cleanly separate the two causes of wasted skills.
References
- Scaling Multi-Agent Reinforcement Learning with Selective Parameter Sharing
- HPS: Hyperspherical Parameter Sharing for Efficient Multi-Agent Reinforcement Learning
- Heterogeneous Skill Learning for Multi-agent Tasks
- Hierarchical Multi-Agent Skill Discovery
- DGPO: Discovering Multiple Strategies with Diversity-Guided Policy Optimization
- HyperMARL: Adaptive Hypernetworks for Multi-Agent RL
- Parameter Sharing with Network Pruning for Scalable Multi-Agent Deep Reinforcement Learning
- Celebrating Diversity in Shared Multi-Agent Reinforcement Learning
- The StarCraft Multi-Agent Challenge
- Google Research Football: A Novel Reinforcement Learning Environment