When thousands of home batteries or electric-vehicle chargers each learn to charge when electricity is cheapest, they all learn the same answer. They then pile into the same cheap hour and create a new peak, a "rebound peak," that can be as bad as the one the tariff was meant to remove. Federated learning trains one shared model across many devices without collecting their private data, and it is increasingly proposed for exactly this kind of control. It makes the problem worse, because averaging every device's model into one global policy also pushes every device toward the same behavior. This note proposes changing what a federated training round carries. Besides model weights, each client sends the load curve its current policy would draw tomorrow. The server learns only the sum of these curves, damps it across rounds, and returns it as a congestion price that every client trains against in the next round.
The point is that in demand response the "moving target" that makes multi-agent learning unstable has a very small dimension. No household needs to know what any other household does. It needs only the total load curve, which sets the price it pays. A federated round already runs a synchronous loop between server and clients. That loop can therefore play a second role as a slow, damped market in which a privacy-preserving estimate of the aggregate load is fixed point by point. Two things follow. Agents learn against the aggregate their peers' current policies will produce, rather than yesterday's realized load. And the desynchronization comes from a cost each agent actually pays, not from an entropy bonus that asks agents to sacrifice their own interest.
Core Claim
Our claim is that federated reinforcement learning for flexible loads should be built as a damped best-response iteration on aggregate load. Each round, client $i$ simulates its current policy on its private calendar to produce a 24-hour load profile $\ell_i^k$ at round $k$. A secure-sum protocol reveals only $\sum_i \ell_i^k$ to the server. The server keeps a smoothed aggregate
$$\bar L^{k+1} = (1-\eta)\,\bar L^{k} + \eta \sum_i \ell_i^k ,$$
where $\eta\in(0,1]$ is a damping step. It broadcasts this aggregate together with the averaged model weights. Each client then trains locally against the price signal
$$p_t = \pi_t + \beta\, \bar L^{k+1}_t ,$$
where $\pi_t$ is the exogenous wholesale price or carbon intensity at hour $t$ and $\beta$ is a congestion coefficient. This coefficient could be a real distribution tariff, or it could be a shadow cost set by the operator. The additive form is the simplest price that makes a hour's cost rise with load; any monotone function would do.
We expect three results. First, plain federated averaging with a price-taking reward will produce rebound peaks that grow with fleet size, while the damped-aggregate version keeps the peak-to-average ratio roughly flat. Second, setting $\eta=1$ (no damping) will reproduce the oscillations predicted for responsive automated agents under real-time pricing. Third, the round-to-round aggregate will converge to a fixed point that is the federated analogue of an aggregative-game equilibrium. That equilibrium is reached without the multi-round schedule negotiation that classical solvers need.
Where This Sits
This work is motivated by a well-documented failure. Magnússon et al. showed that broadcast allocation signals without feedback trigger surges beyond capacity. Jacquot et al. and Wu observed sharp synchronization at the moment off-peak tariffs begin. Roozbehani et al. found that even a small share of highly responsive controllers can destabilize a market. The federated EV controller of Federated Reinforcement Learning for Real-Time Electric Vehicle Charging and Discharging Control trains Soft Actor-Critic agents locally and averages them with FedAvg, and it treats CAISO prices as exogenous. It never asks what happens when the whole fleet runs the averaged policy. That is the gap we fill. The single-station Carbon-Aware EV Charging study warns of the same rebound risk in its own conclusion.
Two multi-agent energy papers already attack herding, and they define the bar we must clear. EB-C-MADRL gives agents the previous day's market aggregates as a stand-in for their peers and adds a bonus for choosing less popular actions. It does this with a single shared network and no privacy model. PRE-MPPO-CBE replaces the one-day lag with a centrally trained forecaster of collective behavior. It adds a KL-divergence "collective behavior entropy" penalty and explicitly leaves incentive compatibility and heterogeneity to future work. We differ from both in three ways. The aggregate we condition on is the one induced by current policies, not one realized under stale policies. It is obtained through federated secure summation rather than central data. And the desynchronizing force is a congestion price that each agent pays, rather than a diversity bonus.
Two older lines of work supply method rather than motivation. The damping-fee scheme of Decentralised Control of Micro-Storage in the Smart Grid proves Lyapunov stability when agents are penalized for changing their profile from day to day. We move that damping out of the agents' cost and into the server's aggregate update, so that agents are not penalized for genuine flexibility. The aggregative-game solver of Generalized Stochastic Dynamic Aggregative Game for Demand-Side Management reaches a unique equilibrium by exchanging schedules with a coordinator over many iterations. We replace that inner negotiation with learning rounds that happen anyway.
From the multi-agent learning side, the survey Dealing with Non-Stationarity in Multi-Agent Deep Reinforcement Learning frames the problem: each learner's environment shifts as its peers learn. It also names the standard fix, a centralized critic that sees all agents' actions as in MADDPG. That fix exposes exactly the data households refuse to share, and it scales poorly to thousands of agents. Our position is that aggregative structure makes such a critic unnecessary, because a critic conditioned on the summed load is enough.
Research Program
We will build a simulator for a residential feeder. Household base load comes from the 200-household Midwest profiles (Muratori 2018), and solar comes from the University of Queensland Gatton irradiance traces, both as used by EB-C-MADRL and PRE-MPPO-CBE. EV arrival, departure, and distance come from NHTS2017, which gives the non-identical commuter, early-bird, and overtime profiles used in the federated EV work. Wholesale prices come from CAISO 2017, and a carbon-signal variant uses the LBNL sessions with CAISO fuel-mix intensity. Each client runs Soft Actor-Critic on its own household. The observation includes the broadcast aggregate $\bar L$, and the reward includes cost at price $p_t$ plus the anxiety penalty for low battery charge at departure from the federated EV paper.
Clients share weights in a personalized way: a common backbone is averaged with FedAvg, and a local head is kept private. This keeps the comparison honest. Personalization by itself is one of the competitors we test, so the aggregate signal must earn its gain on top of it.
The analytical part is modest. For a simplified quadratic-cost version, we will check whether the damped aggregate map is a contraction for $\eta$ below a threshold that depends on $\beta$ and fleet flexibility. This would mirror the damping-fee stability argument. We will then test empirically whether the learned system respects that threshold.
Baselines
The naive reference point is uncontrolled charging (plug in and charge at full power). Next to it sits a greedy price-taker that schedules each household against $\pi_t$ alone. Together they show how large the rebound effect is in our simulator.
The most direct prior method is the federated EV controller of Federated Reinforcement Learning for Real-Time Electric Vehicle Charging and Discharging Control, FedAvg over Soft Actor-Critic with exogenous prices. We reproduce it on our feeder with the same networks, the same number of rounds, and the same tuning budget. That paper reports convergence within about 250 training episodes and does not report aggregate peak. We therefore expect it to win on per-household cost at low fleet size and to lose on peak at high fleet size.
The strongest herding-aware method is PRE-MPPO-CBE. It reports a 38.8% reduction in peak load compared with standard multi-agent approaches on the 200-household data, while EB-C-MADRL reports a 5.15% daily peak reduction and 24.69% lower cost than DQN. These figures were measured against their own baselines, so we will re-run both methods on our feeder rather than compare against the published numbers. We will give them the advantage they assume, centralized access to collective statistics. We count a result as beating them if we reach a comparable or lower peak-to-average ratio at equal or lower mean household cost, while revealing only a summed profile.
We also run the damping-fee controller from Decentralised Control of Micro-Storage, which reports RMS demand-prediction error falling from 31.5 to 14.1 and wholesale volatility falling from 6.6% to 5.3%. It is tuned over the same grid of its damping parameter as we use for $\eta$.
Three ablations isolate the key ingredient. The first feeds clients yesterday's realized aggregate instead of the within-round induced aggregate. The second sets $\eta=1$. The third swaps the congestion price for a federated version of the KL entropy penalty that uses the same securely summed statistics.
Why not just…?
Why not just add random start delays to each charger? Random dithering flattens the boundary spike cheaply and is a real threat on the peak metric alone. It does not respond to where the aggregate actually is, however. It also ignores deadlines, and the early-bird and overtime profiles in the federated EV study are exactly the users it would leave under-charged. Carbon-Aware EV Charging found that the deadline-driven heuristic it tested, earliest-deadline-first, missed most of the carbon savings available. We will report deadline violations and cost alongside peak. If dithering matches us on all three, the learning component is not justified, and we will say so.
Why not just let the utility feed realized demand back into a real-time price? This keeps the learners unchanged. Roozbehani et al. predict that automated responders oscillate under such feedback, and independent learners chasing a price that moves day to day are the textbook non-stationary setting. Our no-damping and yesterday's-aggregate ablations are this alternative in disguise. If they converge as well as the damped variant, our main mechanism is redundant.
Why not just personalize the federated model, since diverse users will diversify naturally? Personalization does address non-identical habits. But commuters who share a schedule still share a cheapest hour, so the herd re-forms within each user type. We test this directly by running personalized FedAvg without the aggregate and reporting peak height within each behavioral profile.
Why not just add the collective-entropy penalty to federated training? This is the obvious combination and our closest competitor. It needs the same herd statistics we compute, and it asks agents to pay for diversity without telling them why. The entropy-penalty ablation answers whether a congestion price does better at the same level of information. The answer would matter for incentive compatibility, the issue the CBE authors left open, even if the peak numbers turn out similar.
Proposed Experiments
The decisive experiment is a penetration sweep. We vary the number of automated households from tens up to the full 200 and replicate profiles to reach 1,000 agents. For each method we plot peak-to-average ratio, maximum feeder load, the height of the spike at the off-peak boundary, mean household cost, deadline violations, and carbon emitted. The hypothesis fails if plain federated Soft Actor-Critic shows no growth in rebound peak with fleet size. It also fails if our method's peak grows at the same rate.
The second experiment is a shock test modeled on the damping-fee robustness study. Midway through training, half the fleet leaves or a cloudy week replaces the solar traces. We count the rounds each method needs to restore its pre-shock aggregate, and we expect the yesterday's-aggregate variants and the damping fee to lag.
The third experiment traces convergence. We plot the round-to-round change in $\bar L$ across a grid of $\eta$ and $\beta$ and check it against the contraction threshold from the analysis.
Finally, we run a privacy check. We attempt membership and schedule inference from the released aggregates at fleet sizes of 20 and 200. We report at what size the summed profile stops identifying individual households, and whether adding calibrated noise to the sum changes the peak results.
References
- Distributed resource allocation using one-way communication with applications to power networks
- Demand side management in the smart grid: An efficiency and fairness tradeoff
- The Optimal EV Charging Strategy in Smart Grid and Insight into Practical Solutions
- Volatility of Power Grids Under Real-Time Pricing
- Federated Reinforcement Learning for Real-Time Electric Vehicle Charging and Discharging Control
- Carbon-Aware EV Charging
- Large-Scale Home Energy Management Using Entropy-Based Collective Multiagent Deep Reinforcement Learning Framework
- Continuous Multiagent Control Using Collective Behavior Entropy for Large-Scale Home Energy Management
- Decentralised Control of Micro-Storage in the Smart Grid
- Generalized Stochastic Dynamic Aggregative Game for Demand-Side Management in Microgrids With Shared Battery
- Dealing with Non-Stationarity in Multi-Agent Deep Reinforcement Learning
- Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments