Heterogeneous-Agent Successor Features: Reusing One Team's Experience Across Changing Goals

Teams of different robots, such as an arm paired with a mobile base or two dexterous hands with different jobs, are now trained well by methods that update one agent at a time and give each agent its own network. These methods have a costly weakness. Their critic, the network that scores how good an action is, predicts a single number tied to one reward function. If the goal changes even slightly, for example "go faster but use less energy" or "cover landmark B instead of A", the critic is wrong and the team is retrained from scratch. We propose giving these one-agent-at-a-time learners a critic that predicts a vector instead of a number. Each entry of the vector is the expected future total of one measurable quantity, such as velocity, energy, or distance to each landmark, and any goal written as a weighted sum of those quantities is scored by a dot product with the weights. Because the math that justifies one-agent-at-a-time updates is linear, it holds separately for each entry of the vector. That means a single batch of experience can train the whole heterogeneous team on many goals at once, and a new goal can be tried with no new training.

The practical claim is that a heterogeneous team pretrained this way should reach good performance on a new weighting of reward terms either immediately or after a small fraction of the environment steps a from-scratch or fine-tuned scalar learner needs. We would also get that transfer without the parameter sharing that current multi-agent transfer methods quietly assume.

Core Claim

The technical starting point is the Multi-Agent Advantage Decomposition Lemma from Heterogeneous-Agent Reinforcement Learning. HARL, the family of methods in that paper, trains each agent with its own parameters, ordered one after another, with monotonic improvement guarantees. The lemma says that the advantage of a joint action taken by agents $i_1,\dots,i_m$ (how much better it is than the team's average behavior) equals the sum of each agent's advantage given the actions already chosen by the agents before it in the ordering. Suppose rewards are linear in a feature vector, $r(s,\mathbf a)=\phi(s,\mathbf a)^\top w$, where $\phi$ holds measurable quantities and $w$ is the goal's weighting, as in Successor Features for Transfer in Reinforcement Learning.. Then the team's action value is $\psi^{\boldsymbol\pi}(s,\mathbf a)^\top w$, where $\psi$ is the successor feature: the expected discounted future sum of $\phi$. Since the lemma is a telescoping identity over value functions, it applies coordinate-wise to $\psi$:

$$\mathbf{A}_\psi^{i_{1:m}}(s,\mathbf a^{i_{1:m}})=\sum_{j=1}^{m}\mathbf{A}_\psi^{i_j}(s,\mathbf a^{i_{1:j-1}},a^{i_j}),\qquad A_w^{i_{1:m}}=\mathbf{A}_\psi^{i_{1:m}}\,{}^\top w,$$

where $\mathbf A_\psi$ is the vector-valued advantage computed from $\psi$ and $A_w$ is the ordinary scalar advantage for goal $w$. Each heterogeneous agent's sequential update can therefore be evaluated for any $w$ from the same vector critic. We build a task-conditioned version of the maximum-entropy sequential learner HASAC (Maximum Entropy Heterogeneous-Agent Reinforcement Learning). In it, each agent $i$ has its own policy $\pi^i(a^i\mid o^i,w)$, and a centralized critic predicts $\psi(s,\mathbf a,w)$ for the joint policy conditioned on $w$. Every stored transition is relabeled with many sampled goal weights during training. We call this HA-SF.

Conditioning the successor features on $w$ addresses a real difficulty flagged in the file. In multi-agent settings, a teammate who adapts to a new goal changes the world from its partners' perspective, which invalidates successor features learned for a fixed team. Here $\psi(\cdot,w)$ is defined for the team as it jointly behaves under $w$, so teammate adaptation is part of what is learned rather than noise to be filtered.

Relation to Prior Work

The closest work is Disentangling Successor Features for Coordination in Multi-agent Reinforcement Learning. Its method, DISSC, uses successor features to split the team value into dynamics and task weights, and uses a learned filter to isolate each agent's contribution for credit assignment. Its agents are homogeneous, its authors leave agent-specific filters for heterogeneous roles to future work, and its goal is better credit assignment on a fixed task rather than transfer across goals. HA-SF differs on all three counts. Credit assignment comes from the exact decomposition lemma rather than a learned filter, policies are unshared, and the target is reuse across goal weightings.

Two further papers show that decoupling rewards from dynamics pays off in multi-agent settings, but both are for uniform agents. The event-based successor representation in Successor Features Based Multi-Agent RL for Event-Based Decentralized MDPs. transfers across reward weights in a subway coverage task, reporting "Nearly 50% improvement" over its baseline on hard settings, but assumes identical agents. The skill graphs of Multi-Task Multi-Agent Reinforcement Learning via Skill Graphs warm-start identical swarm robots from embedded skills.

On the heterogeneous side, HARL, HASAC, and A2PO (Order Matters: Agent-by-agent Policy Optimization) supply the sequential-update machinery we reuse. All three bind the critic to one scalar reward. Composing Task Knowledge With Modular Successor Feature Approximators (MSFA) shows that the features themselves can be discovered by reward regression in single-agent domains. We treat that as an extension, not the core, because it constrains what we can honestly claim: our main results use features built from the reward terms the benchmarks already expose.

Research Program

We would build HA-SF on top of HASAC's off-policy sequential training and change three things. First, the scalar critic becomes a $\psi$-network trained by vector-valued temporal-difference learning, with next-step actions drawn from $\pi(\cdot\mid s',w)$ for the relabeled $w$. Second, each agent's policy is conditioned on $w$. Third, each gradient step samples a set of goal weights from a training distribution over the weight simplex (or a box in weight space), so one replay buffer serves all of them.

The entropy temperature is kept as a vector-valued term with a single fixed coefficient per agent. The HASAC paper flags auto-tuning the temperature across unlike action spaces as fragile, and we do not want goal-dependent temperature to confound results.

At test time on an unseen goal $w'$, there are two modes. In zero-shot mode, the team simply runs $\pi(\cdot\mid o,w')$. It can also apply team-level generalized policy improvement, a rule that picks whichever stored policy the critic predicts is best: it scores the policies for the nearest training weights $w_k$ with $\psi(s,\mathbf a,w_k)^\top w'$ and executes the best one jointly. In few-shot mode, the team continues sequential updates on $w'$ while keeping the pretrained $\psi$ and training data.

We would also analyze when the decomposition-based improvement guarantee survives function approximation error in $\psi$. The expectation is that the per-step error bound scales with $\|w\|$ times the $\psi$ error, since the scalar advantage is a dot product. That predicts transfer should degrade for large-norm or out-of-range weights, and we would test this prediction directly.

Baselines

The naive reference is HASAC trained from scratch on each test goal with a fixed budget. Its converged return also defines the "oracle" used to normalize every other method.

The strongest existing competitor is HASAC pretrained on the training goals and then fine-tuned on the new goal. HASAC is the best reported off-policy heterogeneous learner in the file: it achieved the best performance on 31 of 35 tasks and learned Bi-DexHands policies within 10M steps where MAPPO learned nothing. The file reports no numeric return for HASAC on MAMuJoCo, MPE, or Bi-DexHands. We will therefore rerun it with the released hyperparameters.

HATD3 from HARL serves as a second heterogeneous baseline. For credit-assignment-based successor features, we use a DISSC-style baseline with shared parameters and its learned filter. This tests whether homogeneous successor-feature machinery is enough on heterogeneous teams.

The key ablation is a task-conditioned scalar critic $Q(s,\mathbf a,w)$ trained on exactly the same relabeled data. The relabeled reward is $\phi^\top w$, and every other part of HA-SF stays the same. This isolates the one ingredient we claim matters: predicting successor features and taking a dot product, rather than predicting a scalar value directly.

All methods receive the same pretraining environment steps, the same network sizes, and the same hyperparameter-search budget of 20 configurations each. Evaluation uses 5 seeds, and fine-tuning budgets are equal across methods.

Beating the fine-tuned HASAC means reaching 90% of oracle return on held-out goals in at most one quarter of its fine-tuning steps, with non-overlapping 95% intervals across seeds. Beating the scalar ablation means doing so specifically on goals outside the training weight range. Inside that range, we expect the two to be close.

Why not just…?

Why not just condition a scalar critic on the goal and relabel rewards? This is the real threat. If the scalar ablation matches HA-SF everywhere, the successor-feature part is unnecessary and the contribution shrinks to "multi-goal relabeling for HASAC." Our argument for the vector critic is that linearity in $w$ is built into the model rather than learned. The scalar network has to interpolate over $w$, and it has no reason to extrapolate correctly to held-out weight regions. The held-out-region experiment below settles this. If the gap is absent, we will report the simpler method as the finding.

Why not just fine-tune the pretrained HASAC team on the new reward? Fine-tuning reuses the networks' weights but not the value structure. The critic's scalar output is wrong for the new reward and must be relearned, and the HASAC paper itself notes that changing objectives means retraining. The adaptation curves will show how much of HA-SF's advantage survives against a warm start. A small advantage would suggest pretraining alone does most of the work.

Why not just share parameters across agents, as the existing multi-agent successor-feature methods do? HARL reports that shared-parameter MAPPO "failing to learn" on 17-agent Humanoid, where sequential heterogeneous methods succeed. The DISSC-style baseline on our heterogeneous tasks tests whether that failure carries over to the transfer setting.

Why not just learn from demonstrations? In CH-MARL (CH-MARL: A Multimodal Benchmark for Cooperative, Heterogeneous Multi-Agent Reinforcement Learning), behavior cloning reached up to 48% success on seen layouts but failed on unseen ones. It also needs new demonstrations for every new goal, which is exactly the retraining cost we aim to remove. We therefore do not treat it as a competitive baseline for goal shift.

Proposed Experiments

The primary testbed is Multi-Agent MuJoCo. Its reward is already a sum of terms, which gives natural features: forward velocity, lateral velocity, control cost, and a height or health term. We use HalfCheetah split 6×1, Ant 4×2, and the 17-agent Humanoid, where each agent controls different joints. Goals are weightings over these terms. For example, "run backward cheaply" and "go sideways while staying tall" are held-out goals drawn from regions of weight space that are excluded during pretraining.

The second testbed is a heterogeneous variant of the Multi-Agent Particle Environment's Spread task, with agents of different speeds and sizes. Features are per-landmark coverage and collision counts, and goals set priorities over landmarks.

As a stretch, Bi-DexHands tasks with separable reward terms test whether the approach holds up in high-dimensional manipulation. There we would use MSFA-style learned features, since hand-built features are harder to justify.

Metrics are three. The first is zero-shot normalized return on held-out goals. The second is environment steps to 90% of oracle after a goal shift. The third is cumulative steps over a sequence of eight goal changes, which directly measures retraining overhead.

Three checks would decide the outcome. First, HA-SF should beat the scalar goal-conditioned ablation on out-of-range goals while matching it within range. If it fails this, the vector critic is not the reason for any gains. Second, HA-SF should beat fine-tuned HASAC on steps-to-90% by at least fourfold on Humanoid. That is where heterogeneity and dimensionality are largest, and where an equal result would show the benefit does not scale. Third, transfer quality should fall as $\|w'\|$ grows or as $w'$ moves away from the training distribution, in line with the error-bound prediction. A flat curve would mean our account of why the method works is wrong, even if the method does well.