When a robot team loses a member and gains a replacement with a different body, the newcomer usually learns from scratch, and it pays for that in collisions. The old member's knowledge of where the dangers were is thrown away. That knowledge lives in its cost critic, a network that predicts how much future constraint violation an action will incur. We propose to carry this cost critic across the body change so the newcomer explores safely from its first episode. We also propose a way to decide how much to trust the transferred critic that is not fooled by the teammates, who keep learning and changing their behavior around the newcomer.
The last point is the crux. The closest existing method, QAvatar, learns maps from the target agent's states and actions into the source agent's spaces. It measures how well the source critic satisfies the Bellman equation (the one-step consistency between a value and its successor) on target transitions, and it blends source and target critics in proportion to that consistency. It does this in single-agent settings, and only for reward critics. In a team, the same consistency error also rises when teammates change their policies, even if the body map is perfect. A trust signal built on that error will therefore throw away the transferred safety knowledge exactly when teammates are adapting, which is when the newcomer is most likely to cause a collision. The research file names this conflation as an open gap: no method separates teammate drift from morphology mismatch.
Core Claim
A transferred joint cost critic stays useful for safe exploration in a co-adapting team if its trust weight is computed from a Bellman residual that holds teammate behavior fixed at a reference. Teammate drift is then handled by recomputing the bootstrap with current teammate policies, not by discounting the critic. Concretely, take agent $i$ with a new body, learned maps $\phi$ (target states to source states) and $\psi$ (target actions of agent $i$ to source actions), and a source joint cost critic $Q_C^S(s, a_i, a_{-i})$, where $a_{-i}$ are the teammates' actions. On a target transition with cost $c$, we compute a morphology residual in which the teammates' next actions are drawn from their reference policies $\pi^{\text{ref}}_{-i}$. These are the policies they had when the source critic was trained, which is well defined when only one agent's body changes.
$$\delta_{\text{morph}} = c + \gamma\, \mathbb{E}_{a_i' \sim \pi_i,\; a_{-i}' \sim \pi^{\text{ref}}_{-i}}\big[Q_C^S(\phi(s'), \psi(a_i'), a_{-i}')\big] - Q_C^S(\phi(s), \psi(a_i), a_{-i})$$
Here $\gamma$ is the discount and $\pi_i$ is the newcomer's current policy. The trust weight $\alpha$ in the hybrid cost critic $Q_C = \alpha Q_C^S\circ(\phi,\psi) + (1-\alpha) Q_C^T$ is set from the ratio of $\delta_{\text{morph}}$ to the target critic's own residual, following QAvatar's rule but with this residual in place of the full one. The drift term is the gap between the bootstrap under current and reference teammate policies. It is not used to lower $\alpha$. It is absorbed by using current teammate policies in the actual training target, as a joint-action critic such as MADDPG already does.
The benefit ties into the safety paradox described in Breaking Safety Paradox with Feasible Dual Policy Iteration: once a policy is safe, it stops producing violations, so its cost critic loses information about where the boundary lies. The source team's critic was trained on violations the newcomer should never have to produce. Transfer supplies boundary information that the target cannot safely collect for itself.
Research Program
Each part of the method has a clear origin. Translation and the hybrid-critic machinery, including the normalizing-flow feasibility constraint on $\psi$, come from QAvatar. Constrained optimization uses a Lagrangian with PID damping, which Safety Gymnasium showed reduces multiplier oscillation. The team learner is centralized-training MAPPO-L, the centralized baseline in Scal-MAPPO-L.
The new pieces are three. The first is the transfer of a cost critic rather than a reward critic. The second is the reference-teammate residual. The third is a conservative rule for the period when $\alpha$ is uncertain early in training: the cost used in the constraint is the larger of the hybrid estimate and the target estimate, so poor transfer can only make the newcomer more cautious, never less.
The first stage is diagnostic and needs no new algorithm. We freeze the maps at a known-correct identity (no body change) and vary the teammates' learning rate. We then measure whether QAvatar's original $\alpha$ collapses as teammates learn faster. If it does not, Gap 1 is not real in practice and the proposal reduces to multi-agent cost-critic transfer with QAvatar unchanged. We would report that honestly. The second stage runs the full method on body-change scenarios. The third stage tests how the benefit depends on how far the new body is from the old one.
Baselines
The naive reference point trains the newcomer from scratch inside the team, using MAPPO-L with the same Lagrangian and cost budget and no transfer. It sets the floor for violations accumulated during training. No reported result was found on Lacuna for MAPPO-L's numerical cost on Safe MAMuJoCo. The file states only that Scal-MAPPO-L kept costs below threshold while matching centralized MAPPO-L, so we run this baseline ourselves.
The strongest existing competitor is QAvatar applied per agent. The newcomer uses QAvatar's maps and full-residual $\alpha$ to transfer both reward and cost critics into a joint-action critic, with teammates trained as usual. It is the only method in the file that transfers critics across mismatched continuous spaces and was evaluated in Safety-Gym navigation. For reference, its paper reports reaching the performance threshold with 44% of the environment steps of SAC from scratch, at roughly twice the wall-clock time per step. Beating it means strictly fewer cumulative training-time violations at equal or better final reward. The gap should widen as teammate learning rate increases and vanish when teammates are frozen, since that is what the mechanism predicts.
Two ablations isolate the key ingredient. The first keeps our architecture but uses the full residual for $\alpha$, which amounts to QAvatar with cost transfer. It isolates the reference-teammate residual. The second removes cost-critic transfer and transfers only the reward critic. It isolates whether safety benefits come from transferring cost knowledge at all.
All methods share the same source team checkpoint, the same maps architecture and flow, the same environment-step budget, three-seed hyperparameter sweeps of equal size on learning rates and PID gains, and ten evaluation seeds.
Why not just…?
Why not just freeze the teammates while the newcomer adapts? Freezing removes non-stationarity, and QAvatar then applies unchanged. This is a real threat, and it is run as a baseline. Its cost is that teammates cannot adjust to the newcomer's different speed or reach. In tightly coupled locomotion such as ManyAgent Ant, a frozen team may cap final reward or force a second, unprotected adaptation phase after unfreezing. We measure violations across both phases together. If freeze-then-unfreeze matches our method on total violations and final reward, the disentangling mechanism is unnecessary, and we would say so.
Why not just treat the newcomer as an independent safe learner with teammates as background? Independent learners face a shifting effective environment. MADDPG reports near 0% success for independent learners on cooperative communication, against 84% with a joint critic. Independent Lagrangian multipliers are also known to oscillate (Safety Gymnasium). Independent QAvatar-plus-Lagrangian is included as a baseline to test this directly for the transfer case.
Why not just add a hard safety filter and skip critic transfer? A filter is attractive because it does not depend on the body. However, Safe Deep Reinforcement Learning for Multi-Agent Systems with Continuous Action Spaces reports that hard projections found no feasible action in 56.7% of episodes under disturbances. Its soft variant needs a centralized filter at execution time and a hand-linearized constraint model that must be rebuilt for each new body. We include the soft layer as a complementary baseline and test whether combining it with our method helps further.
Why not just always use the more pessimistic of the two critics? This rule, with no trust weight, is a fair and simple option. It ensures safety but will over-constrain wherever the translated critic is wrong in the pessimistic direction, and early "safe but useless" freezing is a known failure mode (Quanyi Li et al.). It is run as an ablation. We report reward and cost together so that a win on safety alone is not counted as a win.
Proposed Experiments
The main testbed is Safe MAMuJoCo ManyAgent Ant, as used by Scal-MAPPO-L. The source team is trained in one partition. The target replaces one agent's body by changing that agent's leg count through repartitioning (2×3 to 3×2, which changes that agent's state and action dimensions) and by changing gear ratios and link masses. A second testbed is Safety-Gymnasium's partitioned multi-agent Ant. A third is a heterogeneous Multi-Agent Particle Environment variant with collision cost, in which one agent's mass and maximum speed change. For context in the particle setting, the soft safety layer reports over 97% collision reduction relative to MADDPG. Metrics are cumulative cost during training (the primary safety measure), steps to a fixed reward threshold, final normalized reward and cost, and the trajectory of $\alpha$ over training.
The decisive check is a two-by-two factorial design: morphology mismatch (none vs. large) crossed with teammate adaptation (frozen vs. fast). Our $\alpha$ should respond only to mismatch. The full-residual $\alpha$ should also drop under fast adaptation with no mismatch. Training-time violations should track this difference. If our $\alpha$ also drops under pure teammate drift, the reference residual fails to disentangle, and the claim is refuted.
A second check sweeps mismatch magnitude to locate the point where transfer stops helping. At that point, the pessimistic rule should make our method no worse than training from scratch.
References
- Cross-Domain Policy Optimization via Bellman Consistency and Hybrid Critics
- Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments
- Breaking Safety Paradox with Feasible Dual Policy Iteration
- Safety Gymnasium: A Unified Safe Reinforcement Learning Benchmark
- Scalable Constrained Policy Optimization for Safe Multi-agent Reinforcement Learning
- Safe Deep Reinforcement Learning for Multi-Agent Systems with Continuous Action Spaces
- MetaDrive: Composing Diverse Driving Scenarios for Generalizable Reinforcement Learning