Executed Physics Probes as Repair Prompts for Friction and Polymer-Aging Simulators

Large language models now repair ordinary software reasonably well, but they fail on scientific simulation code in a characteristic way. They produce patches that run, pass the repository's tests, and still break the physics: energy that is not conserved, or damage that heals itself. Worse, giving the model the relevant equations as text often makes things worse. The model anchors on the written theory and stops paying attention to what the code actually does. We propose a different way to put domain knowledge into the repair prompt. Instead of pasting equations, we run a small set of physics probes on the buggy code and put the measured results in the prompt. A probe is a reference-free check, meaning it needs no known correct answer. Examples are "total frictional work must equal heat stored plus heat lost," "shrinkage from sunlight never reverses," and "with zero sliding speed there is no frictional heat." The prompt reports which probe failed, at what input, and by how much.

The testbed is two kinds of coupled physics simulators. The first models frictional heating that leads to thermal cracking, as in clutch drums and belts. The second models polymer parts that shrink permanently under ultraviolet light and heat. Neither kind has a trustworthy ground-truth output for most inputs, so a repair oracle that needs the right answer is unavailable. Physical relations like conservation, irreversibility, limits, and scaling are always available. If executed probes beat both equations-as-text and generic retry under the same compute budget, we get a practical recipe for repairing scientific code, where correct outputs are rarely known.

Core Claim

The claim is that domain knowledge helps code repair when it arrives as executed, reference-free invariant violations, and hurts or does little when it arrives as text. Two papers motivate this. SWE-bench Science benchmarks coding agents on 119 tasks from scientific repositories. Its best agent reached 47.90% pass@1, and it documents an "anchoring trap": adding scientific literature to the context made agents fail tasks they had solved without it. Execution-Grounded Agents introduces Oracle-Guided Trajectory Search (OGTS), which lets a model search over repairs using feedback from a deterministic oracle. OGTS raised Llama-3.1-70b from a 58% retry baseline to 84%. That oracle, however, encodes the correct formula or tolerance for each of 50 isolated high-energy-physics scripts. Our proposal differs in exactly that respect. Our probes do not know the right output, only relations the right output must satisfy. They are applied to multi-module coupled simulators rather than single formulas.

The probes contain the same physics a textbook excerpt would. That lets us separate content from delivery. We can give the model identical invariants either as prose or as execution reports and measure the difference directly.

Why this physics regime

The two materials directions supply the invariants, and the bugs worth catching sit where the two directions couple. For frictional heating, the energy budget is the central relation. Experimental Analysis on High-Rated High-Elastic Clutch Coupling ties drum cracking to the ratio of absorbed energy to sliding time; it reports no numeric ratio. The same paper says non-steady heat transfer in that setting still needs better simulation. Advanced Calculation Model of V-Belt Temperature Field attributes over 65% of belt heating to internal bending hysteresis rather than surface friction, so a simulator that drops one source still "runs" but mispartitions heat. Heat leakage modeling for cold compressors gives a scaling law, gas-shear heating proportional to the cube of rotational speed, which a probe can check by doubling the speed and expecting an eightfold rise.

For polymer aging, the key property is irreversibility. Mizyurkin et al. report cumulative, irreversible shrinkage of synthetic ropes outdoors. A common simulator bug models this as reversible thermal contraction, and the part then regains its length when the sun sets. The programmable polyurethane elastomer study predicts strength after heat or UV treatment with coupled Fickian diffusion and DiBenedetto models, at about 3.4% deviation. That supplies a mass-balance invariant for the diffusion part.

The coupling is the gap identified in the file. UV shrinkage changes clearances and contact pressure, which changes frictional heat. Yet the tribology models cited above assume fixed geometry and virgin properties. We treat these sources as the provider of invariants, not as claims we will re-test experimentally.

The energy-balance probe, as one example, computes a normalized residual

$$r_E = \frac{\left|\int_0^T \mu\, p\, v\, A\, dt - (\Delta U + Q_{\text{out}})\right|}{\int_0^T \mu\, p\, v\, A\, dt}.$$

Here $\mu$ is the friction coefficient, $p$ the contact pressure, $v$ the sliding speed, $A$ the contact area, $T$ the simulated duration, $\Delta U$ the change in stored thermal energy, and $Q_{\text{out}}$ the heat lost to the surroundings. The probe fails when $r_E$ exceeds the scheme's discretization tolerance. That tolerance is estimated from the residual's decay under mesh refinement, which the correct reference code shows at its known convergence order.

Research Program

Benchmark. First we build a benchmark, which the file shows does not exist. We will write six reference simulators in Python, each a few hundred to a few thousand lines across several modules, and check each against analytical limit cases. The six are:

  1. a one-dimensional transient thermal model of a clutch drum with a frictional heat flux and a thermal-stress crack criterion;
  2. a V-belt model with both surface friction and hysteresis heating;
  3. a rope aging model with UV- and temperature-driven irreversible shrinkage;
  4. a hygrothermal nylon fatigue model with moisture acting as a plasticizer;
  5. a Fickian diffusion plus DiBenedetto elastomer model;
  6. a coupled bearing-liner model in which UV and moisture shrinkage alter clearance and therefore frictional heating.

Bugs come from two sources.

  • Expert-seeded bugs (about 200). A domain author inserts sign errors, unit mismatches, wrong exponents, lost irreversibility, wrong operator-splitting order at the coupling, and boundary conditions that leak heat.
  • Natural bugs. We have language models implement the same simulators from written specifications and keep every implementation that disagrees with the reference.

Each bug comes with a public test file of the kind real repositories carry, meaning a few smoke and regression tests. It also comes with a hidden validation suite. The hidden suite compares against the reference implementation on randomized inputs and on analytical cases, and it is never shown to any method.

Probe library. Next we build a probe library of about ten templates:

  • conservation of energy and mass;
  • irreversibility and monotonicity in dose;
  • zero-input limits;
  • scaling laws;
  • dimensional consistency under a unit change;
  • symmetry;
  • convergence under mesh refinement.

Instantiating a template for a given code means pointing it at the relevant functions and state variables. We will measure how long humans take to do this, and whether a language model can do it from the docstrings alone.

Repair loop. The repair loop runs the probes on the buggy code and renders failures as short structured reports. A typical report reads: "energy residual 0.31 at v=12 m/s, t>2 s; shrinkage decreased from 0.8% to 0.5% when UV dose was held at zero." The report is added to the prompt alongside the failing public test output. We keep the search procedure OGTS-style, sampling patches, re-probing, and keeping improvements, so that the only thing that changes across conditions is what the feedback contains.

External validity. Finally, we test transfer to the Materials Science and Chemistry tasks of SWE-bench Science. There, probes are instantiated by the model itself, which tells us whether the approach survives outside code we wrote.

Baselines

Naive reference. The model sees the buggy code, the issue text, and the failing public test output, and retries with fresh samples up to the budget. This is the "blind retry" condition of Execution-Grounded Agents, which reported 58% for Llama-3.1-70b on its Scientific Oracle Suite.

Strongest existing method. This is OGTS with a correctness oracle: the hidden validation suite's pass or fail result and its numeric deviation are fed back at every step. It reported 84% for Llama-3.1-70b and saturated at 100% with Claude-3.7-Sonnet on its suite. On repository-scale scientific tasks, the strongest reported agent number is 47.90% pass@1 for Claude-Opus-5 on SWE-bench Science. There is no reported result on Lacuna for repair of frictional-thermal or polymer-aging simulators.

This OGTS condition "cheats": it sees answers no real maintainer has. So it is an upper bound, not a competitor to beat. Success for the proposal means recovering most of the gap between blind retry and this upper bound without reference answers. Concretely, we require closing at least half the gap, with a bootstrap confidence interval over bugs that excludes zero.

Fairness controls. All conditions use the same model family, the same number of model calls, the same number of code executions, and the same wall-clock limit. Probe executions count against the execution budget. Prompt templates are tuned on a held-out one-sixth of simulators, and the tuning budget is identical for every condition.

Key ablation: text versus executed. The ablation that isolates our ingredient gives the model the exact same invariants written as equations and prose, with no execution results. If the executed condition does not beat this text condition, the claim is false.

Test-augmentation baseline. A second baseline uses the model to generate additional unit tests from the code and issue, in the spirit of the test strengthening in SWE-ABS. These generated tests are then used as feedback.

Why not just…?

Why not just paste the governing equations into the prompt? This is the anchoring-trap condition. SWE-bench Science found that auxiliary scientific text sometimes reduced success. Its ablation, though, compared text against nothing, not against executed evidence carrying the same content. Our text-versus-executed ablation holds content fixed. If text matches executed probes on our benchmark, we will report that the anchoring trap does not appear in this regime and the probes add only engineering cost.

Why not just generate more unit tests with the model? This is the real threat. SWE-ABS showed that model-driven test strengthening exposes many wrong patches, with 19.78% of "solved" patches among top agents found incorrect. The risk with generated tests is that they are written by reading the buggy code, so their expected values may encode the bug. Probes avoid this by asserting relations rather than values. We test that directly by counting how often generated tests pass on the buggy code versus how often probes do. If test augmentation matches probes at equal budget, the honest conclusion is that relation-style tests are what matters and the "probe" framing is just a good test-writing prompt. We would then fold the idea into test generation.

Why not just use OGTS with whatever oracle exists? For these simulators there is usually no oracle that knows the right output. That gap is why the clutch study calls for better simulation, and why aging models built on unaged parameters mislead (Evaluation of aging effects on polymers). We include the correctness oracle only as an upper bound, as described in Baselines.

Why not just sample more patches from a stronger model? The best-of-N budget-matched retry baseline answers this. We also report the overfitting rate, meaning patches that pass public tests but fail the hidden suite. More sampling should raise public pass rates faster than hidden ones, while probes should shrink that gap.

Decisive Checks

The primary metric is the hidden-suite pass rate. The secondary metrics are:

  • the overfitting rate, as defined above;
  • probe-level false alarms on correct code, which we measure by running every probe on the six references;
  • cost per repaired bug.

Four results would settle the question.

  • Content versus delivery. Executed probes beat text invariants on seeded bugs and on natural bugs separately. If the gain appears only on seeded bugs, the benchmark is too easy.
  • Coupling bugs. The gain is largest on bugs at the shrinkage-to-friction coupling, since single-physics tests miss those by construction. We will report results by bug category to check this.
  • Model-written probes. Probes written by the model on SWE-bench Science materials tasks keep at least part of the benefit. If the benefit vanishes, the method depends on expert probe authoring, and we will say so.
  • Anchoring on bad probes. Deliberately wrong probes, such as a scaling law with the wrong exponent, should mislead the model less than wrong text does. This tests whether execution evidence is the thing protecting against anchoring or merely correlated with it.