Training Fleet Controllers Along the Battery Frontier

A parking lot full of plugged-in electric vehicles can act like one large battery for the grid. Each car only needs its energy by its departure time, so the operator can speed up or slow down charging to soak up surplus power or shed load. Earlier theory shows this "virtual battery" has a hard limit. The more of the fleet's storage you try to use, the less room you have to charge faster or slower on short notice. For identical cars arriving at a steady rate, this limit has a closed form, and a simple family of scheduling rules reaches it. Real fleets do not look like that. Cars differ in energy needs and deadlines, and they arrive at random. No one knows where the real limit lies for such fleets, or whether any controller gets close to it.

We propose to learn a controller that adjusts the existing optimal scheduling rule over time, and to train it with a curriculum built from the theoretical limit itself. A curriculum here means training on easy cases first and harder cases later. The agent starts on virtual-battery targets that sit well inside the feasible region and moves toward targets on the boundary, while it keeps revisiting the easy ones. The method has two outputs. The first is the first measured limit for heterogeneous, real-data fleets. The second is a test of whether a learned, time-varying schedule pushes that limit beyond the best fixed rule. The curriculum is the tool that makes near-boundary training possible. The limit is both the measure of difficulty and the result we report.

Core Claim

Battery Capacity of Deferrable Energy Demand models each load by its arrival, energy need, deadline and power cap. It proves that any causal scheduler, meaning one that cannot see future grid signals, obeys

$$(\overline{w} + \underline{w} - c)^2 \le 4\,\overline{w}\,\underline{w}\,(1-c).$$

Here $c$ is the normalized storage the fleet promises, $\overline{w}$ is the normalized maximum extra charging rate, and $\underline{w}$ is the normalized maximum curtailment rate. At $c = 1$, one of the two rates must be zero.

Energy Storage From Aggregate Deferrable Demand introduces mixed-slack policies. These rank loads by a blend of two quantities. "Charge slack" is how long a load could keep charging at full power. "Discharge slack" is how long it could stop charging and still meet its deadline. A single weight $\eta \in [0,1]$ sets the blend. The paper proves these policies reach the boundary at its extremes, but only for identical loads, deterministic arrivals and a fixed battery specification.

Our claim has two parts. For heterogeneous, stochastic fleets, a fixed $\eta$ leaves part of the feasible region unused. A policy that sets $\eta$ per load class from the current distribution of slacks can recover that region. Training such a policy directly on boundary targets fails, because the reward is a rare, catastrophic tracking failure. A curriculum indexed by distance to the analytical boundary fixes this.

We define the difficulty of a target $(c, \overline{w}, \underline{w})$ by the margin

$$g = 4\,\overline{w}\,\underline{w}\,(1-c) - (\overline{w}+\underline{w}-c)^2.$$

Training starts at large $g$. It lowers the admitted margin step by step and lets it go slightly negative, because the homogeneous bound is only a proxy once fleets are heterogeneous.

Research Program

Controller design. The controller keeps the structure of the mixed-slack policy and learns only its weighting. At each control step, it observes the requested aggregate power, a histogram of the active loads' charge and discharge slacks, and a short history of the signal. It outputs one $\eta$ per load class, where classes are defined by remaining energy and time to deadline. A fixed dispatcher then serves the requested power in priority order. The action space stays small, and the policy can always fall back to a known optimal rule. This sets it apart from the convex model predictive control (MPC, which re-solves an optimization over a rolling horizon) in Adaptive Charging Networks. That method weighs objectives with static weights and assumes forecasts. Our setting is the causal, worst-case-signal tracking problem that defines the frontier.

Training signal. An episode fails when the fleet cannot follow an admissible signal or a car leaves short of its energy. This is the kind of sparse, unforgiving reward under which Benchmarking Massively Parallelized Multi-Task Reinforcement Learning for Robotics Tasks reports that multi-task reinforcement learning fails without a progressive curriculum.

Curriculum schedule. We follow the mixed design of How does Multi-Task Training Affect Transformer In-Context Capabilities?, in which earlier easy tasks stay in the training mix. In that paper, training only on quadratic functions converged 0% of the time. A mixed curriculum reached 60% success with 1/9th of the data, and a purely sequential curriculum forgot earlier tasks. Those results motivate our design. They also constrain it: a sequential schedule should be expected to fail.

Self-Paced Learning for Latent Variable Models admits samples by their current loss. Its paper notes the method is sensitive to how the admission threshold is annealed. We use it as an automatic alternative to our hand-indexed margin.

Measurement protocol. We estimate the empirical feasible set. For each pair of rates on a grid, we find the largest $c$ the controller sustains at 99% success over held-out admissible signals. These signals include adversarial ones that swing between full charging and full curtailment. Feasible volume is the summary metric.

Environments. The stylized fleet uses the parameters of the mixed-slack paper: 60 kWh needs, 10-hour windows, 18 kW peak, 10 arrivals per hour and about 100 active cars. It runs in a vectorized simulator. The realistic setting draws arrivals and energy requests from ACN-Data, the record of more than 66,000 real charging sessions, and runs them through ACN-Sim. ACN-Sim exposes a Gym interface for reinforcement learning, but that paper trains no agents. In ACN-Sim we can add discrete pilot levels and three-phase limits as a final stress test.

Baselines

Naive references. Least-laxity-first and earliest-deadline-first are the industry defaults. Least-laxity-first equals $\eta = 1$, so the theory predicts its charging headroom collapses. We report these rules' feasible volume as a floor.

Strongest structured baseline: best fixed mixed-slack policy. For every target, we choose $\eta$ by grid search on the test fleet distribution. Its reported reference points are the two emulated batteries in Energy Storage From Aggregate Deferrable Demand. Battery A has 3 MWh, 300 kW charging and 600 kW curtailment. Battery B has 1 MWh, 900 kW and 600 kW. These are shown achievable on the homogeneous fleet. No reported result on heterogeneous fleets was found on Lacuna. We give this baseline as many evaluated configurations as the learned policy uses in tuning trials.

Hand-designed adaptive rule. This baseline sets $\eta$ from the current aggregate state of charge and the sign of recent signals. It gets the same tuning budget as the learned policy.

Robust MPC. This baseline uses the ASA formulation from Adaptive Charging Networks with a worst-case box on the signal. That paper reports profit within 3.6% of offline optimal in idealized settings, but on a profit task, not frontier tracking. No reported result was found on Lacuna for MPC on causal virtual-battery tracking.

Training ablations. All of these use PPO with the same environment steps, seeds and hyperparameter search. We train on uniform sampling over all targets, on a sequential schedule, on a generic curriculum that only ramps signal amplitude, and on a self-paced schedule. The amplitude-ramp ablation is the one that tests whether the frontier geometry matters for training.

What counts as beating a baseline. The learned controller beats the best fixed policy if its feasible volume on ACN-Data fleets exceeds that policy's by more than the spread across five seeds. The curriculum beats its ablations if it succeeds at near-boundary targets ($g$ near 0) where uniform training does not.

Why not just…?

Why not just tune a fixed $\eta$ per target? On homogeneous fleets this is provably optimal at the extremes. We therefore expect, and will check, that learning gains nothing there. Heterogeneity is the only place learning can help. If the tuned fixed $\eta$ matches the learned policy on ACN-Data, we will report that result. The empirical heterogeneous frontier is still a contribution, since the theory cannot supply it.

Why not just use a hand-written adaptive $\eta$ rule? This is the real threat. A rule keyed to aggregate state of charge may capture most of the gain. We give it the same tuning budget. We also check whether the learned policy's advantage concentrates in states where the slack histogram is multimodal, which is information a scalar rule cannot see. If it does not, we will say the simpler rule suffices.

Why not just use MPC? The frontier is defined against unknown, possibly adversarial signals. Forecast-based MPC is the wrong tool for that problem. Deferrable Load Scheduling under Demand Charge finds its block MPC within 0.16% of optimal under perfect forecasts, with performance dropping once arrivals are random. Robust MPC is included as a baseline, so this question is settled empirically rather than assumed.

Why not just train uniformly with lots of compute, or ramp signal amplitude? The evidence from MTBench and the function-class study predicts that uniform training fails on sparse failure rewards. Our uniform ablation tests that prediction here. The amplitude-ramp ablation separates "any curriculum" from "a curriculum indexed by the frontier margin."

Decisive Checks

The first check is a sanity test. On the homogeneous fleet, the learned controller must reach the analytical boundary and never cross it. Crossing it would mean a simulator or evaluation bug, because the bound holds for every causal scheduler.

The second check is the main result. On ACN-Data fleets in ACN-Sim, we compare feasible volume across the learned policy, the best fixed $\eta$, the adaptive rule and robust MPC.

The third check concerns training. We compare success rates at $g \le 0.05$ under the mixed frontier curriculum against the uniform, sequential, amplitude-ramp and self-paced schedules. We expect uniform training to fail at these near-boundary targets, as the cited work predicts.

The fourth check is a stress test. We add discrete pilot levels and three-phase limits in ACN-Sim and measure how much each method's frontier shrinks.