Many organizations would like to train one control policy together without sharing their data or their rules. Examples include utilities running battery fleets, building operators, and robot owners. Each runs its own simulator and has its own private safety limits. Today they face a bad choice. One option is federated reinforcement learning, where agents train locally and send only model updates to a coordinator, which averages them. In the federated methods that respect private constraints, agents learn those limits through soft penalties and break them repeatedly during training. The other option is to put a hard safety filter in front of each agent. A filter of this kind, called a shield, is a small solver that projects any proposed action onto the set of actions known to be safe. It guarantees zero violations, but nothing in federated training uses what the shield knows. Each agent's policy can therefore stay careless and lean on its local shield indefinitely.
This note proposes to use the shield's corrections as the learning signal. Each client's environment wrapper reports how far the shield had to move each action. A local penalty weight, which is never shared, turns that distance into a cost. The shared policy then absorbs every client's private constraints, while no unsafe action is ever executed. The two-sentence version is this: run hard shields locally so training never violates anything, and federate a policy that is penalized for needing the shield. If it works, the averaged policy should rarely trigger any client's shield, should stay safe when the shield is unavailable, and should start a new client with far fewer corrections than a policy trained with shields that stay silent.
Core Claim
The closest prior work splits along the line this proposal crosses.
The paper Federated Reinforcement Learning with Constraint Heterogeneity gives each agent private, non-overlapping constraints. It averages policy parameters globally and keeps a Lagrange multiplier per agent, a learned penalty weight on constraint cost. Its FedPPO variant eventually satisfies the union of all constraints with reward comparable to an omniscient centralized PPO. The satisfaction is only asymptotic, so agents violate constraints in intermediate rounds. This paper motivates the proposal by showing that local multipliers can carry private constraint knowledge into a shared policy. It also supplies the primal-dual machinery we reuse.
Two other papers obtain exactly zero violations from runtime filters, but neither federates learning. CommonPower wraps smart-grid simulators behind the standard Gymnasium step/reset interface and automatically builds model-predictive safeguards that project actions. Its safeguarding costs about 13.5% of training time. Shield Decentralization for Safe Multi-Agent Reinforcement Learning splits a central automaton-based shield into local ones and reports zero violations against more than 115 per 100 episodes unshielded. That method assumes a centralized shield exists to be split, which private constraints rule out.
Our claim is that intervention cost can replace constraint violation inside the federated primal-dual loop. Intervention cost is the distance between the action the policy proposed and the action the shield executed, and the shield can always measure it, even when an actual violation never occurs. This substitution changes three things at once. Training violations become zero by construction. The multiplier still learns, because interventions happen whenever the policy would have been unsafe. And the policy is pushed toward needing no shield at all.
We do not claim that penalizing interventions is new for a single agent. The new parts are its use as the carrier of private, heterogeneous constraints in federated training, and the specific failure it targets. That failure is shield-induced client drift. Clients with different shields generate differently distorted data, and plain averaging hides this from the policy.
Each client $i$ wraps a simulator with a projection $\Pi_{C_i}$ onto its private safe set $C_i$. The policy $\pi_\theta$ proposes action $a_t$, and the shield executes $\tilde a_t = \Pi_{C_i}(a_t)$. Client $i$ locally solves
$$\max_\theta \; J_i(\theta) - \lambda_i \big( D_i(\theta) - \delta \big), \qquad D_i(\theta) = \mathbb{E}_{\pi_\theta}\Big[\textstyle\sum_t \gamma^t \lVert a_t - \tilde a_t \rVert\Big].$$
Here $J_i$ is the discounted return earned under the executed actions, $\gamma$ is the discount factor, $\delta \ge 0$ is a tolerated intervention budget, and $\lambda_i \ge 0$ is client $i$'s multiplier. The multiplier is updated by local dual ascent and is never transmitted. Only $\theta$, or its update, is averaged. For the policy gradient, the shield is treated as part of the environment, so the likelihood term uses the proposed action $a_t$.
Research Program
The first deliverable is an interface, not an algorithm. It is a Gymnasium-compatible shield wrapper that returns, through the standard info dictionary, the proposed action, the executed action, the projection distance, and an infeasibility flag. With these fields, any federated RL algorithm can consume shield information without knowing anything about the solver behind it. The wrapper keeps Gymnasium's split between terminated and truncated (Gymnasium) so that bootstrapping stays correct. For grid tasks it delegates projection to CommonPower's Pyomo safeguards. For navigation tasks it uses closed-form projection: velocity clipping, or pushing the agent's position out of a hazard region.
The second deliverable is the algorithm above, Shield-Cost FedPPO, together with an analysis of where drift comes from. Plain averaging over shielded clients converges toward a policy that is good for the average dynamics. This mirrors the "imaginary environment" result of Federated Reinforcement Learning with Environment Heterogeneity, where the error bound grows with environment diversity. Here, however, the shield changes the effective dynamics differently on every client, so averaging alone has no reason to drive any client's interventions down.
When constraints genuinely conflict at the same state, no single policy can satisfy every client. We will measure how often this happens. As a fallback, we add that paper's private per-environment embedding, so the shared network can condition on client identity without seeing the constraint itself.
The third deliverable is a privacy probe. Gradient Inversion in Federated Reinforcement Learning recovers states and 80–90% of actions from shared policy gradients. It also reports that differential privacy noise strong enough to stop the attack collapses returns. Our updates now encode where the shield fired, so they may leak the geometry of $C_i$. We will run the same style of attack against parameter updates to reconstruct hazard locations, and report the leakage honestly rather than claim privacy we have not measured.
Baselines
The naive reference is independent local PPO behind each client's own shield, with no federation. It shows what collaboration buys. FedHPD reports that independent agents needed about 1200 rounds to reach a CartPole return above 300 versus about 700 with federated exchange. Its independent agents also plateaued near 400 on InvertedPendulum while federated agents exceeded 600.
The strongest competitor on our exact question is shield-in-the-loop FedPPO without any intervention cost. Each client trains behind its shield and the coordinator averages parameters. It also has zero training violations, so it is the real threat. Our method must beat it on the measures that zero violations cannot distinguish: intervention rate, violations once the shield is removed, and cost to adapt to a new client.
The strongest existing federated constrained method is FedPPO with local multipliers on true constraint cost and no shield (Federated Reinforcement Learning with Constraint Heterogeneity). No numeric result is reported in the file for its violation counts or returns; it reports reward parity with the omniscient baseline. The closest single-agent reference point is Safety-Gymnasium, where PPO-Lag cut velocity-task violations by 98% at a 45% reward cost (Safety Gymnasium). To beat FedPPO, we must match its final reward within noise while recording zero executed violations, where it records a positive count.
As an upper reference, we train a centralized shielded PPO that sees all constraints, analogous to the omniscient $PPO_o$.
All methods use the same clients, seeds, number of environment steps, communication rounds, network size, and a common hyperparameter search budget. Two ablations isolate the key ingredient. The first replaces the learned local $\lambda_i$ with a fixed intervention penalty, which tests whether adaptivity matters. The second shares a single global multiplier, which tests whether keeping multipliers private and local matters.
Why not just…?
Why not just put a shield on every client and run ordinary federated averaging? This is the main threat, and it may well suffice in mild cases. The shield already guarantees safety, and the averaged policy may learn to avoid corrections implicitly, because corrected actions sometimes lose reward. Our hypothesis is that this implicit signal is weak whenever projection is nearly reward-neutral. In that case the policy keeps proposing unsafe actions and depends on the shield. The decisive tests are the shield-off evaluation and the held-out-client intervention rate. If plain shielded averaging matches us on both, the intervention cost is unnecessary, and we would report that.
Why not just use federated primal-dual training and accept early violations? In grid and robot settings, a violation during training is a damaged feeder or a collision, not a logging event. The constraint-heterogeneity paper's guarantees are asymptotic. We will count executed violations per round for FedPPO against our method's zero. We will also check whether the shield's guidance shortens the time for the multipliers to settle.
Why not just share the constraints with a central coordinator and build one decentralized shield? That is the route of shield decentralization (Shield Decentralization). It requires the constraints, and a shared labeling of the environment, to be pooled. Pooling is exactly what private operational limits forbid, in the same way the virtual power plant work keeps customer battery data behind a hierarchy (A Scalable, Distribution Network-Aware, Customer Privacy-Preserving Framework for Operation of Virtual Power Plants). We include the centralized shielded learner only as an upper bound, not as a deployable option.
Why not just add differential privacy noise to protect the constraint geometry? The gradient inversion paper reports that noise sufficient to block reconstruction degrades returns to unacceptable levels. We therefore treat leakage as something to measure first. Defenses are out of scope unless the probe shows hazard geometry is easy to recover.
Proposed Experiments
The first testbed is Safety-Gymnasium navigation and velocity tasks, split into eight clients. Each client receives a different private hazard layout or velocity limit, which mirrors the distinct prohibited regions of the constraint-heterogeneity paper. Each shield is a closed-form projection.
The second testbed is CommonPower on the Kerber Landnetz 30-bus network. It is split into six clients that differ in battery capacity, thermal limits, and Simbench and When2Heat load profiles. Each client has its own Pyomo voltage and capacity safeguard. We test whether solver overhead stays near the reported 13.5%.
The primary metrics are four. We record return, and executed violations during training, which must be zero for shielded methods. We record intervention rate and mean projection distance per round, and violations when the trained policy is evaluated without its shield.
The decisive checks are three. First, on clients held out from training, Shield-Cost FedPPO should start with a markedly lower intervention rate than shielded averaging and keep reward comparable to FedPPO. Second, with shields removed, its violations should approach FedPPO's converged level, while the shielded-averaging policy violates often. Third, the learned-multiplier ablation should beat the fixed penalty on clients whose constraints are looser or tighter than average.
A further diagnostic tracks the reward cost of safety. If it approaches the 45% drop seen for PPO-Lag in Safety-Gymnasium, the intervention budget $\delta$ is too strict, and we will report the trade-off curve over $\delta$ rather than a single point. The gradient-inversion probe reports how accurately hazard positions can be recovered from the transmitted updates, compared with shielded averaging. This shows whether the intervention cost leaks more than the baseline does.
References
- Federated Reinforcement Learning with Constraint Heterogeneity
- CommonPower: A Framework for Safe Data-Driven Smart Grid Control
- Shield Decentralization for Safe Multi-Agent Reinforcement Learning
- Gymnasium: A Standard Interface for Reinforcement Learning Environments
- Federated Reinforcement Learning with Environment Heterogeneity
- Gradient Inversion in Federated Reinforcement Learning
- FedHPD: Heterogeneous Federated Reinforcement Learning via Policy Distillation
- Safety Gymnasium: A Unified Safe Reinforcement Learning Benchmark
- A Scalable, Distribution Network-Aware, Customer Privacy-Preserving Framework for Operation of Virtual Power Plants