Counterexample-Guided Refinement of LLM-Written Robust Dispatch Models

Language models can now write optimization code that runs, but in power-system scheduling, code that runs is the easy part. A dispatch model that schedules generators, batteries and reserves a day ahead under uncertain wind and solar can solve cleanly and still be wrong. It might quietly let tomorrow's commitments peek at tomorrow's weather. It might drop a battery efficiency term, or guard against impossible weather such as zero solar output at a clear noon, and so buy far too much reserve. Compiler errors and unit tests do not catch these mistakes, and they are exactly the mistakes a generalist model makes when it lacks domain knowledge.

We propose to give the model a different kind of feedback. Instead of tracebacks, it receives a concrete counterexample: a realistic weather scenario, drawn from held-in renewable data, under which its schedule forces load shedding or a voltage violation, or under which it paid for reserve that realistic weather never required. The model then revises its formulation. In two sentences: an independent evaluation harness fixes the model's day-ahead decisions, replays realistic scenarios against them, and returns the single most damaging scenario together with a physical explanation of the damage. We test whether this scenario-level feedback repairs modeling errors that traceback-based self-debugging and scalar-fitness search cannot repair.

Core Claim

The claim is narrow. When a language model writes a two-stage robust dispatch model, feedback in the form of a concrete, physically explained worst-case scenario will raise the share of formulations that are both valid and economical well beyond what three cheaper sources of feedback achieve: execution errors, a scalar out-of-sample cost score, or automatically adding the bad scenario to the model without any edit by the language model.

A two-stage robust model is one in which "here-and-now" commitments, such as unit commitment and reserve, are fixed before uncertainty is revealed, and "wait-and-see" recourse, such as battery dispatch and redispatch, adapts afterward. The method borrows its central idea from column-and-constraint generation, the classical algorithm that repeatedly finds a worst-case scenario and adds it to the master problem. We move that loop up a level. In our version, the worst case is used to correct the program's modeling rather than only its numbers.

Why This Gap, and Why Now

Three findings from prior work motivate the proposal. LLaMoCo shows that GPT-4 Turbo has a 41.6% code error rate when writing optimization code and falls back on default gradient solvers. Its remedy is one-shot instruction tuning, and it explicitly rejects iterative loops as too costly. TheoremQA shows that pasting the relevant theorem into the prompt does not help models apply it. This suggests that domain knowledge has to arrive as instance-level consequences rather than as definitions. CodeTransOcean shows that traceback-driven self-debugging plateaus quickly, from a success rate of 48.57% before any repair round to 52.57% after three rounds. Do Users Write More Insecure Code with AI Assistants? documents the related danger: without an independent check, code that works but is flawed passes, and users feel confident about it.

Two systems come closest to our setting, and both stop short of it. Accelerated GPHH uses an AI pair programmer to speed up a stochastic scheduling simulator by 85%. Its authors say a human navigator was required to keep the logic correct, and the AI never touched the scheduling model itself. $A_2$DEPT evolves whole solver programs with a self-healing dependency loop, but only on deterministic instances, and it reports that "partial migration" bugs survive repair. In both systems the feedback is execution status or a scalar objective. Neither gives the model a reason why its program is wrong.

On the optimization side, two papers supply methods we reuse. AGRO learns a variational autoencoder over historical renewable traces and searches its latent space for worst cases. This removes the "ghost contingencies" of box and polyhedral uncertainty sets and cuts planning cost by 8% on New England expansion planning. We reuse AGRO as one way to generate counterexamples. Robust Energy Management for Microgrids With High-Penetration Renewables identifies the physical components a credible microgrid model must contain, including battery depth-of-discharge and elastic load shifting. These components define our task specifications. Neither of these lines of work involves code synthesis.

Research Program

The first artifact is a task suite. We will write roughly 60 natural-language specifications of day-ahead robust dispatch problems on the IEEE-33 bus distribution network. Scenarios come from DDRE-33, which contains 2,000 single-node wind and solar series and 300 jointly correlated wind-solar scenarios with weather-regime labels. The tasks vary in their components: batteries with efficiency and depth-of-discharge limits, elastic loads, reserve markets, ramp limits, and carbon cost. Each task comes with a reference two-stage formulation written by a domain expert and solved by classical column-and-constraint generation over linearized radial power flow.

DDRE-33's scenarios are split by weather regime into three sets. An adversary pool is visible to the feedback loop. A tuning set is used for model selection. A held-out test set, drawn from disjoint regimes, is used only for final scoring.

The second artifact, and the core of the method, is the evaluation harness. It does not trust the program's own reported objective. It extracts the first-stage decisions, freezes them, and for each scenario re-solves recourse in a separate optimization problem written by the harness. It then audits the result with a full AC power flow solve for voltage and line limits. Freezing the first stage catches a common modeling error that runs silently: indexing day-ahead variables by scenario, which lets commitments see the future.

From the adversary pool, the harness returns the scenario that maximizes realized cost plus a penalty for violations. The search can run over the discrete pool or by AGRO-style gradient ascent in a learned latent space. Along with the scenario, the harness returns a short structured explanation, for example "bus 18, hour 13, 0.4 MW unserved; battery at depth-of-discharge floor since hour 11." It also returns a signal for the opposite failure: when reserve stays unused under every realistic scenario, it reports the cost of that over-procurement. This two-sided signal is what separates the method from plain worst-case feedback, which would only push the model toward more conservatism.

The third artifact is the refinement loop. The model receives the specification, its current program, and the counterexample report, and it edits the formulation. We will run the loop on one open-weight code model and one frontier API model. Each run has a fixed budget of K harness calls. The loop keeps a rollback checkpoint, in the manner of Fang et al., so that it can recover when an edit causes a regression.

The fourth component is a controlled diagnostic. We take the reference formulations and seed them with a taxonomy of known modeling bugs: scenario-indexed first-stage variables, a box uncertainty set where a budget set is specified, battery efficiency applied on the wrong side, a missing ramp constraint, and wrong signs in the power balance. For each bug type and each feedback type, we measure the rate at which the loop repairs the bug. This shows which kind of feedback fixes which kind of knowledge gap, a question no prior work answers.

Baselines

The naive reference point is deterministic dispatch on a point forecast, with no robustness. It tells us how much any robust formulation is worth on DDRE-33. The ceiling is the expert reference formulation solved by column-and-constraint generation. Both are run on the same held-out scenarios through the same harness.

The first prompting baseline is one-shot generation with the relevant formulas and physical constraints included in the prompt. This is the "theorem augmentation" condition that TheoremQA found ineffective for general STEM problems. We expect it to fail similarly here, and testing it rules out the cheapest fix.

The second baseline is traceback self-debugging in the style measured by CodeTransOcean, which reports a success rate of 52.57% after three rounds on its own translation benchmark. It receives the same K-call budget as our loop. Its feedback is limited to execution errors and solver status codes, so infeasible and unbounded results count as feedback but no scenario is supplied.

The strongest competitor is $A_2$DEPT-style evolutionary program search, with the program tree, simulated-annealing acceptance, and maintenance loop as described in $A_2$DEPT. Its fitness is the harness's scalar score on the adversary pool, and it gets the same total number of harness calls. $A_2$DEPT reports a 9.8% reduction in mean normalized optimality gap over earlier LLM heuristic-design methods on deterministic combinatorial benchmarks. No reported result was found on Lacuna for any LLM method on robust dispatch or on DDRE-33. For one-shot optimization code, the strongest numbers on file are from LLaMoCo: a 41.6% code error rate and a 74.2% optimization performance score for GPT-4 Turbo, against roughly 5.5% and 81.8% for LLaMoCo-S. Those were measured on non-robust continuous problems, so we treat them as context, not as a target. All baselines use the same base models, the same scenario splits, and the same budget of harness calls and tokens. Any prompt tuning is done on the tuning set only.

Two ablations isolate the key ingredient. The first, scalar-only, keeps the loop but replaces the counterexample report with the scalar score. The second, automatic augmentation, adds the counterexample scenario to the program's scenario set mechanically, as column-and-constraint generation would, with no edit by the language model. Automatic augmentation can fix numbers but cannot fix modeling bugs. The gap between it and the full method therefore measures what the language model's edit contributes. We also run random-scenario feedback, which supplies a scenario without the adversarial search. The proposal beats the field if three conditions hold. On held-out regimes, the full method must reach a markedly higher rate of formulations that are both valid and close to the reference's expected cost (within 5%) than scalar-fitness evolution at equal budget. It must beat automatic augmentation on the seeded-bug diagnostic. And the advantage must persist on the open-weight model.

Why not just…?

Why not just have an expert write the model and run classical column-and-constraint generation? When an expert is available, that is the right answer. On our 33-bus tasks it is the ceiling we compare against, not a competitor. The proposal targets the step the expert currently performs, which is turning a specification into a correct two-stage formulation. Accelerated GPHH shows that this kind of modeling work is the expensive, human-gated part. We do not claim speedups over Neur2RO-style solvers. A generated formulation could be handed to such a solver.

Why not just evolve programs against a scalar out-of-sample cost, as $A_2$DEPT does? This is the real threat. With enough calls, selection pressure alone might find valid formulations. The scalar-only ablation and the equal-budget $A_2$DEPT baseline settle the question directly. If scalar fitness matches counterexample feedback at equal budget, the claim fails, and we will report that. Our prior expectation that it will not match comes from the seeded-bug diagnostic. A scenario-indexed first stage looks better under in-sample scalar scoring, not worse, so scalar fitness has no gradient toward fixing it unless the evaluation freezes the first stage. Separately, we test whether the freezing harness alone, without explanations, closes the gap.

Why not just self-debug on tracebacks? The failures we care about raise no errors. CodeTransOcean already shows that traceback repair plateaus within three rounds even on ordinary code, and the security study shows that code which runs while being flawed slips through. Our diagnostic reports a per-bug-type repair rate for traceback feedback. We predict it will be near zero for bugs that do not crash, such as an over-conservative uncertainty set or wrong battery efficiency.

Why not just fine-tune on optimization code, as LLaMoCo does? No corpus of two-stage robust dispatch formulations exists. LLaMoCo's 6,000 problems are continuous test functions, and fine-tuning cannot provide instance-specific physical diagnosis. Fine-tuning is complementary, not a rival. Verified trajectories from our loop are precisely the training data such tuning lacks. We will test, as a secondary result, whether a model tuned on those trajectories narrows the gap in one shot.

Decisive Checks

All metrics are measured on held-out DDRE-33 weather regimes, never on the adversary pool. The primary metric is the fraction of tasks whose final program passes the frozen-first-stage AC audit on every test scenario, together with its expected cost relative to the expert reference. Secondary metrics are the over-procurement cost against the reference, the per-bug repair rates, and the number of harness calls needed to reach a valid formulation.

The single most informative outcome is the gap between the full method and automatic augmentation on the seeded-bug suite. If the language model's edit adds nothing beyond mechanically adding the bad scenario, then the knowledge-gap story is wrong, and the useful contribution reduces to the harness itself. We would report that harness as a domain verifier for LLM-written optimization code, a gap the research file flags as unfilled.

A final transfer check asks whether the approach generalizes beyond dispatch. We run the same loop on the robust knapsack and capital-budgeting instances from Neur2RO and compare against their best-known values. Neur2RO reports reaching within roughly 2% of those values.