Scoring Synthetic Grid Scenarios by How Well They Tune a Feedback Controller

Power grids increasingly run on controllers that use measurements instead of models. These controllers repeatedly take one small optimization step using live voltage and current readings. The approach is called online feedback optimization (OFO): rather than solving a full planning problem inside a computer, the controller nudges the actuators a little at a time and lets the physical grid reveal the consequences. OFO is robust to a wrong grid model, but it is not free of tuning. Its step size and sampling period have to be chosen before deployment, and they are almost always chosen on synthetic renewable and load scenarios. Those scenarios come from generative models that are judged by image-style or forecasting-style statistics. No one checks whether they lead to the same controller settings that real traces would.

We propose to measure synthetic data quality by its operational consequence. A generator is good if a controller tuned on its output performs, on real traces, as well as a controller tuned on real traces. We also propose a cheap score meant to predict this quantity without running the full tuning loop. The score compares synthetic and real disturbances only in the frequency band where the tuned controller is sensitive. If this works, grid operators gain a defensible way to say how much they can rely on synthetic scenarios. They also gain a measured exchange rate: how many synthetic days are worth one real day for tuning.

Core Claim

OFO tracking error under a moving optimum depends mainly on how fast the optimum drifts relative to the controller's own speed of convergence. For a controller that contracts toward the current optimum at rate $\rho<1$ per step, the steady tracking error scales roughly as the per-step drift divided by $1-\rho$. That form follows from summing a geometric series of contractions against bounded drift.

The consequence is that the property of a synthetic scenario that matters for tuning is not its marginal distribution. It is the spectrum of its fluctuations over the band of frequencies the closed loop cannot reject. A generator can match marginals and score a low Fréchet distance while getting that band wrong, and it will then choose the wrong step size. We therefore claim two things. First, tuning-transfer regret is the right ground-truth measure of synthetic scenario quality for OFO. Second, a fidelity score that weights the spectral mismatch by the controller's disturbance-to-error sensitivity will rank generators by that regret much better than distributional metrics do.

Let $\theta$ be the controller hyperparameters (step size, sampling period). Let $H_\theta(\omega)$ be the linearized closed-loop gain from disturbance to tracking error at frequency $\omega$. Let $S_{\text{real}}$ and $S_G$ be the power spectra of real and generator-$G$ disturbance traces. The score is

$$D_w(G)=\max_{\theta\in\Theta}\int |H_\theta(\omega)|^2\,\big|S_{\text{real}}(\omega)-S_G(\omega)\big|\,d\omega ,$$

where $\Theta$ is the candidate tuning range. The ground truth it should predict is the regret

$$R(G)=J_{\text{real}}(\hat\theta_G)-J_{\text{real}}(\hat\theta_{\text{real}}),$$

where $\hat\theta_G$ is the setting chosen by tuning on $G$'s scenarios, $\hat\theta_{\text{real}}$ is the setting chosen on real training traces, and $J_{\text{real}}$ is closed-loop cost plus a constraint-violation penalty on held-out real traces.

Relation to Prior Work

The motivation comes from two places.

On the control side, Tuning of Online Feedback Optimization for setpoint tracking in centrifugal compressors tunes step size and sampling period jointly on a simulated compressor and reports up to 87% better tracking than steady-state-based manual tuning. That shows tuning matters a great deal. However, it tunes against a simulator without asking how simulator error moves the chosen settings, and that question is our subject.

On the synthetic data side, On the Equivalency, Substitutability, and Flexibility of Synthetic Data measures a roughly 30:1 synthetic-to-real exchange rate for multi-person tracking. It also finds that behavioural complexity, not photorealism, drives downstream value: multi-group scenes raised the tracking score called HOTA from 68.8 to 73.5, while simple scenes hurt. We borrow its idea of exchange rate and substitutability. We move it from offline perception training to closed-loop controller tuning, where "downstream" means a dynamical system's stability and constraint satisfaction.

Our hypothesis that temporal structure in the band the controller cannot reject matters more than marginals is a direct analogue of its "behaviour over appearance" finding. Is this Model for Real? Simulating Data to Reveal the Proximity of a Model to Reality supplies a related methodological stance. It judges a simulator by whether the parameters fitted on its data land where real-data parameters land. Our regret $R(G)$ is the control analogue: does the tuning optimum land in the same place?

The generators we evaluate come from Privacy-Preserving Spatiotemporal Scenario Generation of Renewable Energies: A Federated Deep Generative Learning Approach. That paper trains a federated GAN (Fed-LSGAN) on NREL wind and solar data and scores it by Fréchet distance (FID) and the continuous ranked probability score (CRPS). It never connects those scores to any controller, and closing that loop is the gap we address.

The OFO controllers come from Experimental validation of feedback optimization in power distribution grids (dual-ascent Volt/VAr control on hardware), Real-time Curative Actions for Power Systems via Online Feedback Optimization (projected-gradient curative control on IEEE 39-bus), and Online Feedback Optimization for Transmission Grid Operation (mixed-integer quadratic steps on the Unicorn 7019 system). All three validate on simulated scenarios without examining them.

One line of work constrains the proposal. The SYSLAB experiment found that OFO with an all-ones sensitivity matrix still reached feasible voltages within 12% of optimal cost, and the 39-bus work found convergence under wrong topology. This suggests that static model error barely matters. We take that seriously: the proposal predicts that synthetic fidelity matters through temporal drift, not through static mismatch, and one experiment tests exactly this split.

Two other papers sit nearby but are not the method. SIPDO: Closed-Loop Prompt Optimization via Synthetic Data Feedback puts a synthetic generator in a loop with an optimizer to find failures. It is adjacent, but it generates hard cases for text prompts rather than measuring whether synthetic data transfers. Lyapunov-Stable Adaptive Control for Multimodal Concept Drift builds its drift from hand-made perturbations. It illustrates the untested-synthetic-drift problem we want to measure, not solve.

Research Program

The work builds an evaluation harness with three parts.

The first part is a pool of disturbance generators for wind and solar feed-in. It contains the GAN family from the Fed-LSGAN paper (Fed-LSGAN, centralized DCGAN, standard GAN), a seasonal autoregressive model, a block bootstrap of real days, and Gaussian-noise-perturbed real traces. It also contains two diagnostic surrogates built from real data. Phase-randomized traces keep the real spectrum exactly but Gaussianize the tails. Time-shuffled traces keep the marginal distribution exactly but destroy the spectrum.

The two surrogates are the instrument that separates our hypothesis from its rivals. If spectrum is what matters, the shuffled surrogate should produce large regret despite perfect marginals, and the phase-randomized surrogate should produce small regret. If tails matter more than we think, the phase-randomized surrogate will also produce large regret.

The second part is a set of OFO controllers on simulated grids driven by these disturbances. The main plant is IEEE 39-bus with projected-gradient OFO. The second plant is a distribution feeder modelled on the SYSLAB setup with dual-ascent Volt/VAr control and the 0.95–1.05 p.u. voltage band. The Unicorn 7019 mixed-integer controller is included if the grid data is accessible to us.

The third part is a tuning procedure. It is a fixed-budget search over step size and sampling period, in the style of Zagorowska et al., run identically on each generator's scenarios.

For each generator and controller we compute $R(G)$, $D_w(G)$, and every competing metric. We also compute an exchange-rate curve: real-only tuning is repeated with $k$ real days for increasing $k$, and we find the $k$ whose regret matches tuning on $N$ synthetic days. The ratio $N/k$ is the exchange rate. It is reported separately for drift-dominated conditions (fast ramps) and mismatch-dominated conditions (wrong sensitivities, slow disturbances). This directly tests the split between static mismatch and temporal drift that the OFO literature leaves open.

The real reference traces are the NREL Washington wind and solar series: 32 wind sites and 12 solar sites at 5-minute resolution. We split them by date into generator-training, tuning, and held-out evaluation. One caveat applies: NREL series are partly modelled products themselves. Wherever SYSLAB-style measured feeder data is available, we repeat the final ranking on it.

Baselines

The naive reference point is a controller tuned with steady-state reasoning and no scenario data. It uses the largest step size that the static stability condition permits, at the default sampling period. This is the "manual tuning" against which Zagorowska et al. report their 87% improvement on the compressor. Any generator whose tuned controller does worse on held-out real traces than this default has negative value, and we report how many generators fall there.

The real-data oracle tunes on the full real tuning split. It defines zero regret, and the exchange-rate curves are anchored to it.

The strongest existing quality measures for scenario generators are FID and CRPS, as used by Li et al. On NREL data they report FID of 4.446 for Fed-LSGAN, against 4.879 for centralized DCGAN and 7.057 for a standard GAN. We use that ranking as the reference ranking. We compute FID, CRPS, and maximum mean discrepancy on the same held-out real traces, with the same number of synthetic samples as our score. Our score beats them if its Spearman rank correlation with $R(G)$ across the pool exceeds theirs by a margin outside the bootstrap confidence interval. In addition, it must flag the time-shuffled surrogate as bad, which a marginal-matching metric cannot do. No reported result was found on Lacuna for the correlation between FID or CRPS and any controller-level outcome, so the baseline correlations are measured here.

The ablation that isolates our key ingredient is the unweighted spectral distance: the same integral with $|H_\theta|^2$ removed. The ablation also includes a distribution distance on first differences, meaning ramp rates. If either matches $D_w$, the controller weighting is not doing the work, and we will report that.

Why not just…?

Why not just trust OFO's robustness and skip evaluating synthetic data? The SYSLAB all-ones result and the wrong-topology result make this a real threat for static model error, and our mismatch-dominated condition may well confirm it with near-zero regret across generators. The evidence against extending it to temporal structure is that tuning itself moves performance a great deal: Zagorowska et al. report up to 87% on a compressor. In addition, Model-Free Nonlinear Feedback Optimization reports large practical tracking errors when gradients are noisy. The drift-dominated condition settles the question. If regret stays small there too, the honest conclusion is that synthetic fidelity barely matters for OFO tuning, and we would publish that.

Why not just tune directly on real traces? Where many real days exist, this is correct, and the oracle baseline is exactly this. The question is how much real data is needed. The exchange-rate curve answers it. If $k$ is tiny (a handful of days), synthetic data adds little and we will say so. The interesting regime is new sites, privacy-restricted data of the kind that motivated federated generation, and rare high-ramp days.

Why not just compute CRPS or MMD on ramp rates instead of a controller-weighted spectrum? This is the most serious alternative, because increments carry much of the drift information. It is included as an ablation rather than argued away. Our expectation, which may be wrong, is that unweighted increment metrics over-penalize high-frequency errors the controller filters out, and under-penalize slow, low-frequency errors near the closed-loop bandwidth. If the increment metric ranks generators as well as $D_w$, the contribution narrows to the regret benchmark and exchange rates. That is still new, but it is smaller.

Why not just use domain randomization over generator outputs? Randomizing scenario parameters during tuning tends to pick conservative step sizes. We run tuning on a randomized mixture of all generators as an extra arm, and compare its held-out real cost with that of the best single generator and the oracle.

Proposed Experiments and Decisive Checks

The first decisive check is the surrogate pair on the 39-bus controller. We predict large regret for the time-shuffled traces and small regret for the phase-randomized traces. If the phase-randomized traces show large regret, tails matter, and $D_w$ must be extended with a term for exceeding constraint margins before it can be trusted.

The second check is the cross-generator rank correlation of $D_w$, FID, CRPS, MMD, and the ablations against $R(G)$. It is computed separately on the 39-bus and feeder controllers, and on Unicorn 7019 if the data is available. It is reported with bootstrap intervals over held-out days.

The third check is whether the FID ranking from Li et al. (Fed-LSGAN better than DCGAN better than standard GAN) survives under regret. A reversal would be the clearest single demonstration that distributional quality and operational quality differ.

The fourth check is the pair of exchange-rate curves for drift-dominated and mismatch-dominated conditions. These give the first measured synthetic-to-real ratio for tuning a feedback controller, and the condition-dependent comparison maps where OFO's model-free robustness actually covers for poor synthetic data.

If hardware access allows, a final transfer check would run the controller settings chosen by the top-ranked and bottom-ranked generators on a SYSLAB-type feeder. It would count violations of the voltage band.