Counterfactual Frost: A Contamination-Resistant Test of Whether Language Agents Understand Heat-Exchanger Fouling

When frost builds up on a heat pump's outdoor coil, it adds an insulating layer between the metal and the air. Heat transfer drops, airflow chokes, and the machine eventually has to stop and melt the ice, which costs energy. Deciding when to defrost (and, in boilers, when to blow off soot) is a small, well-posed engineering decision with real money attached. Language models are increasingly asked to give advice or write control logic for decisions like this. We do not know whether they reason from the physics or recite textbook rules such as "defrost every hour." Static question banks cannot tell these apart, because the textbook rules are in the training data.

We propose a simulator-backed test with three parts. First, every scenario exists in two versions: a factual one, and a counterfactual one where the stated frost-growth law is deliberately altered so that memorized rules give wrong answers. Second, each scenario is probed at four levels, from reading the plant state up to scheduling defrosts. Third, models write hard scenarios for one another and solve each other's scenarios in a round-robin. The central claim is that the gap between factual and counterfactual performance, measured separately at each level, shows where memorized engineering knowledge overrides evidence given in the prompt. The cross-play matrix (which model wrote a scenario versus which model solved it) adds a check that random procedural generation cannot provide: whether a model can solve the scenarios it wrote itself.

Core Claim and Where It Sits

Three bodies of prior work motivate this design without covering it. Dynamic generation has shown that static scores mislead. DyVal builds reasoning problems on the fly from random graphs. Models that score well on static math, such as WizardMath-13B and phi-1.5, scored nearly zero on these generated problems. MLAgentBench found 100% agent success on pre-cutoff tasks but near-zero success on post-cutoff ones. Both stay in abstract or ML-workflow domains. Neither can ask whether a model's physical intuition is memorized, because neither has a physics where the "right answer" can be changed on purpose.

Hierarchical diagnosis has shown that flat scores hide where failures happen. AECBench found models above 95% on engineering terminology that could not pass 60/100 on document creation. GAMEBoT scored intermediate game reasoning against programmatic ground truth, with GPT-4o averaging 0.52. GAMEBoT also observed that Claude 3.5 Sonnet forfeited winning moves to obey strategic instructions. That rigidity is close to what we want to measure: a prior overriding the evidence in front of the model. AECBench is static and exposed to contamination, and GAMEBoT's subproblems are fixed game rules. We borrow their method of scoring each level against a programmatic oracle, but our oracle is a physics simulator whose laws we can edit.

Competitive arenas have used opponents to keep test content fresh. ZeroSumEval pits models against each other so that test content keeps changing. The Red Alert RTS benchmark ranks models by round-robin Elo, with Claude-Sonnet-4.5 at 2136. Its authors acknowledge that models probably saw strategy guides during training. These arenas report terminal outcomes only. We take the round-robin idea but apply it to scenario authorship rather than head-to-head play, because defrost control is not naturally adversarial.

The physical side constrains the design. Fouling work evaluates on fixed plant logs: the economizer studies of Shi et al. 2019 and Shi et al. 2021, and the pipe-fitting soft sensor of Jarmatz et al.. Control benchmarks such as PC-Gym and the I4B heat pump simulator from Constrained RL for Safe Heat Pump Control do not model frost growth at all. We therefore have to build the frost model ourselves and keep it simple enough that ground truth is exact.

Research Program

The simulator is a reduced-order frost model for an evaporator coil. Frost thickness $\delta(t)$ grows at a rate set by the gap between the humidity of the incoming air and the saturation humidity at the coil surface, and by a surface factor that represents coating wettability. Frost conductivity $k_f$ rises as the frost densifies. The overall heat transfer coefficient follows

$$\frac{1}{U(t)} = \frac{1}{U_{\text{clean}}} + \frac{\delta(t)}{k_f(t)},$$

where $U_{\text{clean}}$ is the clean-coil value. Airflow falls as $\delta$ narrows the fin gaps, and a defrost costs a fixed energy plus downtime. Qualitative trends are calibrated to the experimental literature:

Because the dynamics are low-dimensional, a dynamic-programming solver gives the exact optimal defrost schedule for any parameter set. This follows the Hamilton-Jacobi-Bellman approach used for soot blowing (HJB soot blowing). Every scenario therefore has a verifiable answer, which addresses the gap the file notes between dynamic generation and verifiability.

The counterfactual arm keeps the simulator's structure but rewrites one stated law in the prompt, and the simulator obeys the rewritten law. Examples:

  • frost growth that rises with surface temperature;
  • a coating that speeds frosting;
  • frost whose conductivity falls as it densifies.

Real-world heuristics give confidently wrong answers in these worlds. The factual and counterfactual versions of a scenario are otherwise identical: same numbers, same prompt template, same answer format.

Each scenario is probed at four levels, each scored against simulator ground truth. Level 1 asks the model to read the state: current frost thickness, airflow, or coil temperature from a telemetry snippet. Level 2 asks for the direction of a relationship, for example whether raising humidity raises or lowers the thermal resistance after one hour under the stated law. Level 3 asks for a calculation, such as the time until $U$ falls below a threshold. Level 4 asks for a full defrost schedule over a 24-hour weather trace, scored as regret against the dynamic-programming optimum, in units of delivered heat per unit energy. The levels mirror the progression in AECBench, and the per-level accuracies are what let us locate a failure.

Scenario authoring is a round-robin. Each model, given the simulator's parameter interface, writes scenarios (weather traces, coil parameters, optionally a counterfactual law) intended to be hard for others. A scenario is admitted only if the simulator accepts it as physically valid and the optimal schedule beats a fixed timer by a set margin; this rule excludes degenerate traps. Every model then solves every admitted scenario, producing an author-by-solver matrix of regret and per-level accuracy. The diagonal (models solving their own scenarios) tests whether a model's generative knowledge of the physics matches its ability to act on it.

To check that findings carry beyond our simplified frost model, we repeat the four-level probes in a second deposit domain: boiler economizer soot blowing. Here the scenario parameters come from the fouling-increment model of Shi et al. 2019 rather than from our frost simulator.

Baselines

The naive reference for control is a fixed-interval timer: defrost every 60 minutes, or for boilers, the plant's original 8.33-hour accumulation and 0.67-hour blowing cycle. We run it on every scenario. Any agent that cannot beat it at Level 4 has not used the scenario information at all.

The control ceiling is the dynamic-programming optimum. Defining regret against it makes this ceiling zero by construction. For the boiler domain, the closest real-world comparison is the particle-swarm schedule of Shi et al. 2019. They report a 30.96% increase in heat exchange capacity per unit time over the fixed schedule, with $t_d$ falling from 8.33 to 7.62 hours and $t_b$ from 0.67 to 0.46 hours. We reimplement their model so that our oracle and their optimizer can be compared on the same instances. For the frost domain, no reported result was found on Lacuna for learned defrost scheduling.

A trained reinforcement-learning controller, soft actor-critic configured as in PC-Gym, gets the same simulator and a fixed training budget. It shows what a non-language learner achieves, and whether counterfactual worlds are hard in themselves or only hard for models with priors. The RL agent is retrained in each world, so it should show no factual–counterfactual gap. If it does show one, the counterfactual scenarios are simply harder, and we would have to correct for that.

The strongest competitor on the evaluation question is DyVal-style random procedural generation with flat scoring. Scenarios are drawn uniformly from the parameter space and each model receives one regret number. We give this baseline the same number of scenarios and the same inference budget per model as the round-robin. Our design beats it if it produces more stable model rankings, measured by Kendall's tau between disjoint scenario batches, at equal cost, and if its level-wise gaps predict something flat regret does not. One target is which models fail when they are moved from the factual arm to the counterfactual arm. A second strong competitor is regret-maximizing search: an evolutionary search over simulator parameters that looks for instances where a reference model does worst, with an equal compute budget.

Three ablations isolate the ingredients. Flat scoring alone isolates the four-level hierarchy. Dropping the counterfactual arm isolates the memorization measurement. Replacing peer-authored scenarios with search-generated ones isolates authorship.

Why not just…?

Why not just use a static question bank of frost and fouling problems, scored by level, as AECBench does? A static bank cannot separate recall from reasoning, because its answers can be memorized. AECBench itself is built from licensure exams that are likely in training data. The counterfactual arm is our direct test of this concern. If factual and counterfactual Level 2 and Level 3 accuracies match for every model, then memorization is not distorting these tasks, and a cheaper static bank would suffice. We would report that outcome as a negative result.

Why not just generate scenarios at random and skip the round-robin? This is a real threat. DyVal shows that random generation already defeats contamination in formal domains. Random draws may also already be hard enough, since the counterfactual arm alone defeats memorized rules. The head-to-head with the random-generation baseline at equal cost settles it. If random scenarios give equally stable rankings, we keep the round-robin only for its diagonal self-consistency measurement. If that measurement also adds nothing, we drop the round-robin.

Why not just search for adversarial instances with an optimizer instead of letting models write them? For finding hard instances, search may well win. It does not tire, and it targets regret directly. What search cannot provide is the author-versus-solver structure. In particular, it cannot tell us whether a model that can write a scenario where another model's frost intuition fails can avoid that same failure itself. We compare the two methods on hardness and on discriminability. We claim an advantage for peer authorship only on the diagonal measurement.

Why not just evaluate models as controllers with flat regret, as PC-Gym evaluates RL agents? Flat regret tells us that a model's schedule is poor but not why. The cause could be misreading the state, getting the direction of an effect wrong, an arithmetic error, or poor planning. GAMEBoT's 0.52 intermediate score for GPT-4o is the kind of signal flat outcomes hide. If Levels 1–3 fail to predict Level 4 regret, the hierarchy has no diagnostic value. We test this explicitly below.

Proposed Experiments

We evaluate the models named in the file: GPT-4o, Claude-Sonnet-4.5, Gemini-2.5-Flash, DeepSeek-V3.2, Qwen-3-Max, and Llama-3-70B. Each runs with and without a code-execution tool, since a model that writes a simulator call could sidestep its priors. Each model authors 50 admitted scenarios. Every scenario is posed in both its factual and counterfactual version, and every cell of the 6×6 matrix receives all four levels of probes. Rank stability comes from bootstrap resampling over scenarios.

The first decisive check is whether the hierarchy is valid. Level 1–3 accuracies on a scenario should predict Level 4 regret on that scenario beyond what model identity alone predicts. If they do not, the levels are artifacts of how we decomposed the problem, the risk the file flags as rigid decomposition.

The second check locates contamination. We expect the factual–counterfactual gap to be near zero at Level 1, largest at Level 2, where models assert the textbook direction of an effect against the stated law, and to carry into Level 4 regret. If the gap instead appears only at Level 3, the problem is arithmetic rather than memorization, and our interpretation would be wrong.

The third check compares peer authorship with random generation and parameter search on ranking stability per unit compute, and reports the diagonal-minus-off-diagonal gap for each model.

The fourth check is transfer. Model rankings and level profiles from the frost simulator should predict rankings on the boiler soot-blowing scenarios. If they do not, our results are specific to our toy physics rather than evidence about engineering reasoning.