Large language models can now fix code by running it, reading the error, and trying again. This works when the computer can say clearly what is wrong, such as a compiler error or a failed unit test. It breaks down for scientific simulation code, where the most damaging bugs produce no error at all. Take a phase-field solver, a program that simulates how crystals, particles and grain boundaries form as a material solidifies. It can run to completion and draw a plausible-looking microstructure while quietly creating mass from nothing or letting its free energy rise. Nothing crashes, so there is nothing for the model to react to.
We propose to give the repair loop the checks a careful computational materials scientist would run. These include whether total solute is conserved, whether the free energy falls monotonically, whether a flat interface travels at the speed theory predicts, whether rotating the initial condition rotates the result, and whether coexisting phases settle at the compositions the thermodynamic database prescribes. Each failed check is reported back to the model, tied to the part of the code that most likely caused it. The application we target is hard-particle nanocomposite coatings formed by in-situ reaction during laser cladding. In this setting the physics changes from one alloy system to the next, so the simulation code itself, not just its parameters, has to be rewritten.
Core Claim
Execution-feedback loops plateau after roughly three rounds. Yan et al. report this, and the file describes the later rounds as cyclic or degenerate edits. Our hypothesis is that, for physics simulation code, the plateau comes mainly from feedback that carries too little information, not from limits on what the model can reason about. Once code compiles and runs, crash and test-pass signals stop distinguishing a correct solver from a subtly wrong one.
Physical invariants are properties any correct solver must satisfy, whatever the exact answer is. They keep supplying graded, localizable signal after the code runs. Two such properties are the mass balance of a conserved composition field $c(\mathbf{x},t)$ and energy dissipation for a gradient-flow model with free energy functional $F$:
$$\frac{d}{dt}\int_\Omega c\,d\mathbf{x} = 0 \ \ \text{(no-flux boundaries)}, \qquad \frac{dF}{dt}\le 0,$$
where $\Omega$ is the simulation domain. The concrete prediction has three parts. Feeding back a battery of such checks, each tied to a specific term in the code, should keep raising success rates beyond iteration three on phase-field extension and repair tasks. It should beat crash-only feedback, unit-test feedback, and image-based feedback. It should also sharply cut the rate of "silent wrong" solvers, meaning code that the agent believes is correct but that disagrees with a held-out expert reference.
What is new is the feedback oracle, not the agent architecture. Prior refinement loops in engineering domains pass back compiler traces (MEIC, with ModelSim on Verilog), simulator logs (the ns-3 multi-agent system), profiler hotspots (POLO), or waveform images and scalar circuit metrics (AnalogCoder-Pro). None of them checks continuous-field conservation laws or thermodynamic consistency, which is exactly where SciCode shows frontier models fail on scientific code.
Why This Application
The materials side supplies a case where code must change, which keeps the proposal from collapsing into parameter tuning. In the (Ti, Nb)(C, B)/IN625 ultra-high-speed laser cladding study, the same process produced very different physics. The exothermic Ti–B₄C reaction released about 670 kJ and slowed cooling, which widened the bonded interface from about 4 to about 24 µm. Selective loss of B₄C shifted the stoichiometry toward unplanned carbide fractions. The resulting composite was softer than plain IN625 (240 vs 300 HV0.2) yet wore roughly ten times more slowly.
A simulation that captures any of this needs a reaction heat source coupled to the temperature field, a precursor-loss term, and extra phases. Each of these is a structural code change. Zhang et al.'s phase-equilibrium surrogate work makes the same point from the solver side. Moving to multiphase reactions requires reformulating the model, and their 20.39× speedup required hand-integrating a LibTorch network into C++ code.
Microstructure-aware Bayesian design motivates the gap from the opposite direction. It improves parameter search by looking at intermediate microstructure, but it treats the phase-field code as fixed. We take the code as the thing being changed and keep the parameters as inputs.
Research Program
We will build three artifacts.
A task suite (PF-Extend). The suite contains phase-field coding tasks with expert-written reference solvers. It covers three kinds of task. The first is extensions, such as moving a working binary Al–Cu solidification solver to ternary Ni–Al–Cr, adding a third precipitating phase, or adding a reaction heat source and a volatile-species sink to a temperature-coupled model of reactive cladding. The second is integrations, such as replacing a Newton equilibrium solve with a supplied surrogate. The third is seeded-bug repairs, built the way MEIC built its 178 seeded register-transfer-level errors.
Equilibrium targets come from the Al–Cu, Ni–Al–Cr and CMSX-4 systems used by Zhang et al.. Annealing targets come from the Mg₂Sn$_x$Si$_{1-x}$ setting of the Bayesian design paper, where those data or settings can be reconstructed. The reactive-cladding tasks are reduced two-dimensional models inspired by the IN625 system. We do not claim they predict its measured numbers.
An invariant oracle library. The library implements conservation, dissipation, analytic interface-velocity and Gibbs–Thomson tests, rotation and translation invariance (to catch grid-induced anisotropy), mesh and time-step convergence order, and equilibrium partition checks against the thermodynamic data. Each check is paired with a non-triviality check, so that freezing the dynamics cannot satisfy dissipation trivially. Each check also reports which equation term or function its violation implicates. We find the implicated function by re-running the check with one term zeroed or replaced by a reference implementation.
A refinement loop. The loop follows the MEIC pattern of a generator agent, a scoring agent and rollback to the last snapshot that scored best. The only thing we vary is what the scorer sees.
The analysis has two parts. The first is the success-versus-iteration curve out to twelve rounds, to test whether invariant feedback moves the plateau that Yan et al. observed. The second is a Goodhart audit: we measure how often agents learn to pass the invariants while still disagreeing with the hidden reference. Srivastava et al. warn that weak verification oracles let iterative prompting degrade code quality. Our invariants are a stronger but still incomplete oracle, and measuring this leakage is part of the result, not a footnote.
Baselines
The naive reference point is single-pass generation with the governing equations and a worked binary example in the prompt. SciCode tells us what to expect from this regime. With no background text, o1-preview passes 7.7% of main problems, Claude 3.5 Sonnet 4.6% and DeepSeek-Coder-v2 3.1%, against about 28% on isolated subproblems (SciCode). We will run the same backbone models single-pass on PF-Extend, sampling as many candidates as the loop is allowed to execute and keeping the best by the agent's own choice.
The strongest existing method is a MEIC-style loop with compiler and runtime-log feedback, a scorer agent and rollback. MEIC reports a 78% functional-error fix rate with GPT-4 on its RTL benchmark, against 66% for retrieval and fine-tuning baselines (MEIC). For phase-field code repair, no reported result was found on Lacuna, so we will establish that number ourselves.
We will also run a stronger version of this baseline in which the scorer sees the visible unit tests shipped with each task. Separately, we will run an AnalogCoder-Pro-style variant that feeds rendered microstructure images and summary scalars (phase fractions, interface positions) to a multimodal model (AnalogCoder-Pro).
Every arm uses the same backbone, the same cap on solver executions and tokens, the same rollback policy, and the same prompt-tuning budget on a shared development split. Success is always judged by the hidden expert reference. That reference checks composition fields, phase fractions and interface kinetics within a pre-registered tolerance, and none of the arms ever sees it.
Two ablations isolate the key ingredient. One returns only a global pass/fail over all invariants, which tests whether localization matters. The other returns invariant results without the non-triviality pairing, which tests gaming.
Beating the strongest baseline means a higher hidden-reference success rate at matched execution budget, together with a success curve that is still rising after round three while the baseline has flattened. Beating it on the agent's own score alone does not count.
Why not just…?
Why not just hand-write one parameterized solver and run Bayesian optimization over its inputs? For tasks where the physics is fixed, that is the right tool, and we will not compete there. Our tasks are defined by structural changes, such as new phases, new source terms or a swapped equilibrium routine, which a parameter vector cannot express. That is exactly the case the cladding study's shifting reaction regime creates. A useful pairing would be to have our loop produce the solver and BO search its parameters.
Why not just give the agent unit tests against reference outputs? In real research on a new alloy system no reference output exists, which is the reason to simulate in the first place. Even so, this is a serious competitor, so the visible-test arm is run as described above. If it matches invariant feedback on hidden-reference success, our claim reduces to "invariants are a reference-free substitute for tests." That is still useful, but narrower, and we would report it that way.
Why not just show the model pictures of the microstructure? AnalogCoder-Pro shows that visual feedback helps on circuits. Many phase-field bugs, though, such as a few percent of solute drift, a wrong partition coefficient, or first-order instead of second-order convergence, are invisible in an image. Grid anisotropy, on the other hand, may be quite visible. This is the alternative most likely to rival ours on some bug classes. Breaking results down by seeded-bug category will show where images suffice and where they do not.
Why not just sample many more candidates instead of iterating? If the bottleneck is feedback quality, not search, then best-of-n with equal compute should do as well. The single-pass arm with matched execution count tests this directly. If invariant-filtered best-of-n matches the iterative loop, then the oracle is doing all the work and the iteration is not, and we would say so.
Proposed Experiments
The decisive experiment is the matched-budget comparison of the four feedback arms (crash-only, visible tests, images, invariants) on PF-Extend, run with two backbones of different size. MEIC found that a smaller model inside its loop beat a larger model used alone (64% vs 62% syntax fixes), and we will test whether the same holds here. The prediction fails if the invariant arm's success curve plateaus at round three like the others, or if its gain over visible tests falls within run-to-run variance across five seeds.
The second check is the Goodhart audit. Among solutions that pass every invariant, we measure the fraction that fail the hidden reference, broken down by task family. If this fraction is high for the reactive-cladding tasks, the invariant set is too weak for multiphase reactive physics, which is itself a finding about where verification has to grow.
Third, as an out-of-suite transfer test, we run the invariant-feedback loop on the condensed-matter and materials subproblems of SciCode that involve differential-equation solvers. We report main-problem pass rate against the published single-pass numbers above.
Finally, as a qualitative case study, the best agent-extended reactive-cladding model is checked against one directional trend from the IN625 study: adding the exothermic reaction term should lower the local cooling rate and widen the region that remelts and bonds. We treat agreement as a check that the model is physically sensible, not as quantitative validation. We expect the 4-to-24 µm magnitude only in direction, not in value.
References
- CodeTransOcean: A Comprehensive Multilingual Benchmark for Code Translation
- MEIC: Re-thinking RTL Debug Automation using LLMs
- Toward Generative 6G Simulation: An Experimental Multi-Agent LLM and ns-3 Integration
- POLO: An LLM-Powered Project-Level Code Performance Optimization Framework
- AnalogCoder-Pro: Unifying Analog Circuit Generation and Optimization via Multi-modal LLMs
- SciCode: A Research Coding Benchmark Curated by Scientists
- (Ti, Nb)(C, B)/IN625 In-Situ Reactive Coating Prepared by Ultra-High-Speed Laser Cladding: Interfacial Characterization, Residual Stress and Surface Wear Mechanisms
- Phase equilibrium prediction for multicomponent alloys in phase field simulations using machine learning method
- Microstructure-aware Bayesian materials design
- Do Users Write More Insecure Code with AI Assistants?