Fast-charging controllers for high-energy lithium cells, especially those with silicon in the anode, are increasingly trained with reinforcement learning (RL). In RL, a controller learns by trial and error to maximize a numeric score called the reward. In these battery controllers the reward is hand-written. It weights charging time against penalties on simulated internal damage, such as the growth of the surface film that forms on the anode, particle cracking, and lithium metal plating. Those internal quantities are exactly what battery simulators get most wrong. A controller trained to minimize simulated plating is partly learning to exploit the simulator's plating model. When it moves to a real cell, it fails in a way that neither retuning the weights nor randomizing the simulator reliably fixes.
We propose replacing the hand-written damage terms with a learned per-cycle damage score that has two properties. First, it is calibrated on real cells: we fit it so that, summed over each real cell's life, it accounts for that cell's measured cycle life. Second, it reads only quantities that can be extracted from terminal voltage curves, both on hardware and in simulation. Because the reward is the same function of the same observable features in both worlds, the controller can no longer score well by gaming simulator-only internal states. We test whether this one change narrows the gap between simulated and real cycle life more than the standard remedies do.
Core Claim
The closest work, Ageing-Aware Deep Reinforcement Learning for Adaptive Fast Charging, trains a deterministic policy-gradient controller (DDPG) inside the Doyle-Fuller-Newman electrochemical model, the standard physics model of a lithium cell. It is implemented in the open-source simulator PyBaMM and parameterized for an LG M50T cell. The model includes four coupled ageing mechanisms, and the reward is hand-composed from charging time and degradation penalties. The paper offers no principled way to choose the weights. It runs entirely in simulation and assumes constant temperature. In that setup the reward and the sim-to-real gap are the same problem, because the reward reads the simulator's least trustworthy outputs.
Aging matrix visualizes complexity of battery aging across hundreds of cycling protocols motivates our fix. It cycled 359 commercial NCA/graphite + SiOx 21700 cells under 207 fast-charge protocols, with lifetimes from 44 to 994 equivalent full cycles. It extracted four interpretable health indicators by fitting differential voltage curves: positive-electrode capacity $Q_{PE}$, negative-electrode capacity $Q_{NE}$, lithium inventory $Q_{Li}$, and polarization resistance $R_p$. It also found that early changes in $R_p$ predict end-of-life capacity better than early capacity does. That paper is purely diagnostic and trains no controller. We turn its observables into a reward.
Let $z_k = (Q_{PE}, Q_{NE}, Q_{Li}, R_p)$ be the indicators at diagnostic checkpoint $k$, and let $u_k$ summarize the charging actions applied since the previous checkpoint. We learn a nonnegative damage function $h_\phi$, a small neural network with parameters $\phi$. Its fitting condition is that, for every real training cell $i$ with checkpoints $1,\dots,K_i$ up to end of life,
$$\sum_{k=1}^{K_i} h_\phi(\Delta z_{i,k}, u_{i,k}) \approx 1 .$$
This is a learned version of linear damage accumulation (Miner's rule from fatigue analysis): each checkpoint interval consumes a fraction of the cell's life. The controller's reward between checkpoints is then
$$r_k = -\tau_k - \lambda\, h_\phi(\Delta z_k, u_k),$$
where $\tau_k$ is charging time in the interval. The scalar $\lambda$ converts minutes into fractions of life. Rather than tuning $\lambda$, we report the whole tradeoff curve between charging time and life. In simulation, $z_k$ is computed by running the same differential-voltage fitting on simulated voltage curves, not by reading internal states.
Energy density enters as the operating regime. Silicon-containing anodes store far more charge through alloying reactions that involve several electrons per host atom; Advanced High Energy Density Secondary Batteries with Multi‐Electron Reaction Materials cites 3579 mAh/g theoretical for Si. They also swell and crack. This makes degradation more coupled and harder to model, which is where we expect simulator-internal rewards to mislead most.
Research Program
We build three things. The first is the damage model $h_\phi$, fit on a cell-level split of the aging-matrix data. Its main hazard is the problem Guided Cost Learning reports for learned costs: they stay accurate only near the data and extrapolate badly. We address this in three ways. We constrain $h_\phi$ to increase with $|\Delta R_p|$, $-\Delta Q_{Li}$ and $-\Delta Q_{NE}$, a monotonicity constraint in the spirit of that paper's regularizers. We fit an ensemble of models. During training we penalize the policy where the ensemble members disagree.
The second is the controller. We use the paper's parameterization of the "Mars" charging profile, an initial current $I_0$ and a decay coefficient $K$, re-chosen at each checkpoint as the cell ages. We train it with the same DDPG setup as the ageing-aware paper, so that only the reward changes.
The third is a gap testbed, since we cannot claim free access to a cycler. The training simulator is PyBaMM's single-particle model with electrolyte, run at constant temperature with nominal degradation parameters. The stand-in for the real cell is the full Doyle-Fuller-Newman model with coupled thermal dynamics and held-out perturbations of the side-reaction rates for film growth, plating and cracking. Real data then enters at two checkpoints. One is reward validation on held-out real cells, described below. The other is a small physical cycling study of the finalists. We treat the latter as a necessary confirmation, not an afterthought.
Baselines
The naive reference point is constant-current/constant-voltage charging (CC-CV) at the rate the aging-matrix data identify as typical. It needs no learning. It anchors whether any learned controller is worth having.
The most direct competitor is the ageing-aware DDPG controller with its hand-composed reward, reimplemented with the same network, training budget and observation set. For fairness we give it the same total number of simulator episodes and the same tuning budget for its reward weights as we spend fitting $h_\phi$. I found no reported cycle-life number on Lacuna for the ageing-aware DDPG controller, so we reproduce it rather than quote it.
The strongest sim-to-real competitor is that same hand-written reward plus domain randomization, meaning training across simulators with randomly varied physical parameters. We randomize the side-reaction rates and diffusion coefficients over ranges chosen from stress-test sweeps, in the style of DrEureka. DrEureka uses an LLM to write reward code, uses simulator sweeps to find the parameter ranges where a policy still works, and then sets randomization from them. It reported 34% higher forward velocity than human-tuned settings in quadruped locomotion. Its rewards still read simulator state, and it has never been applied to electrochemistry.
The strongest non-RL competitor is the safe Bayesian-optimization MPC of Safe Learning-Based Optimization of Model Predictive Control. MPC (model predictive control) repeatedly plans a short horizon of charging currents against a cell model. That paper tunes the MPC's cost with Bayesian optimization constrained to stay safe and reports up to 410 seconds faster charging to 80% state of charge than standard MPC, with 15% violating exploration iterations versus 40% for unconstrained Bayesian optimization. Those numbers are for a single cycle on an equivalent-circuit model, which approximates the cell with resistors and capacitors. We run it in our testbed with the same episode budget and report its lifetime outcome.
The key ablation keeps everything about our method but feeds $h_\phi$ the simulator's true internal states instead of voltage-derived features. A second ablation drops learned damage and uses the sparse end-of-life signal directly. Beating the baselines means reaching a better tradeoff curve between charging time and life in the gap testbed, and a smaller gap between predicted and realized life, at matched charging time.
Why not just…?
Why not just run Bayesian optimization over fixed charging protocols directly on real-cell data, skipping simulation? This is the most serious threat, and the aging-matrix dataset makes it cheap. Its limitation is that a fixed protocol cannot react as a cell ages from full health to 60%. Adaptive throttling in late life is precisely what the ageing-aware paper found prevents the sudden capacity drop caused by plating. We settle this empirically. We pick the best fixed protocol according to $h_\phi$ itself, evaluate it in the testbed, and send it to physical cycling alongside our controller. If it matches the adaptive controller, the honest conclusion is that the learned damage model is the contribution and RL is unnecessary. We would report it that way.
Why not just add domain randomization to the hand-written reward? Randomization widens the range of worlds the controller has seen. It does not change the fact that the score is computed on simulated internal states. DrEureka itself notes that wide randomization makes controllers overly conservative. In our setting that means slow charging. The domain-randomization baseline and the internal-state ablation separate the two effects. If the voltage-only reward wins with or without randomization, the observability choice is doing the work.
Why not just tune MPC constraint margins, as in Learning Model Predictive Control Parameters via Bayesian Optimization for Battery Fast Charging? That paper learns a voltage safety margin or a resistance parameter to remove 4.2 V violations under a deliberate 50% mismatch between model and cell. Its tuning objective is still a hand-written time-plus-penalty score over single charges. Voltage safety is not lifetime. A charge that respects every limit can still plate lithium cumulatively. Including the safe-tuning MPC baseline with lifetime as the outcome measures that difference directly.
Why not just use the real cycle life as a sparse reward? Hundreds of cycles separate an action from its consequence, so credit assignment is hopeless at feasible sample budgets. The sparse-reward ablation quantifies how hopeless.
Proposed Experiments
The first decisive check needs no hardware. On held-out aging-matrix cells, we test whether summed $h_\phi$ ranks protocols by realized cycle life better than three alternatives: the summed hand-written penalty evaluated on PyBaMM replays of the same protocols, and the random-forest predictor from the aging-matrix paper. We measure ranking with Spearman correlation and life error in equivalent full cycles. If $h_\phi$ cannot beat the simulator-based penalty here, the premise fails and we stop.
The second check is the gap testbed. We train every controller on the simplified simulator and evaluate on the thermal full model with held-out side-reaction perturbations. We report the tradeoff curve of minutes to 80% charge against cycles to 80% health. The primary metric is the gap between simulator-predicted and testbed life at matched charging time. A further diagnostic is whether hand-written-reward controllers, unlike ours, show low simulated plating but high testbed plating, which is the signature of exploiting the simulator.
The third check is physical. We cycle a small set of SiOx-containing 21700 cells under four policies: CC-CV, the best fixed protocol, the domain-randomized hand-reward controller and ours. We use enough replicate cells per policy to resolve differences that the aging-matrix spread (44–994 cycles) suggests are large. We judge the outcome on early-life $R_p$ trajectories, which the aging-matrix paper shows are predictive, and on capacity at a fixed cycle budget.
References
- Ageing-Aware Deep Reinforcement Learning for Adaptive Fast Charging of Li-ion Battery Considering Coupled Degradation mechanisms
- Aging matrix visualizes complexity of battery aging across hundreds of cycling protocols
- Advanced High Energy Density Secondary Batteries with Multi‐Electron Reaction Materials
- Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization
- DrEureka: Language Model Guided Sim-To-Real Transfer
- Safe Learning-Based Optimization of Model Predictive Control: Application to Battery Fast-Charging
- Learning Model Predictive Control Parameters via Bayesian Optimization for Battery Fast Charging