Language models now write physics simulations: projectile demos, cloth and fluid scenes, PDE solvers. The usual way to check such code is to run tests and report how much of the code they executed. For physics code that check fails in a specific way. Most of a simulation is one unbranched update loop, something like "position plus velocity times timestep." A single short test touches every line of that loop and earns full coverage. Yet the bugs that matter, such as energy that slowly drifts, an integrator that blows up when springs are stiff, or gravity that depends on mass, appear only in particular physical conditions and only over many timesteps.
We propose to replace "which branches did the tests touch" with "which physical regimes did the tests visit." A regime is described by dimensionless quantities such as the timestep-to-stability ratio, the stiffness ratio, the Reynolds number, and the rollout length. We pair this measure with oracles that need no reference program, because the correctness checks come from physics itself: conservation laws, symmetry relations, and the requirement that errors shrink at the expected rate as the timestep is refined. The resulting verifier can label LLM-written simulation code as correct or buggy when no trusted solution exists. We test whether it is accurate enough to select and train better code generators.
Core Claim
For LLM-generated simulation code, branch coverage barely predicts whether a test suite catches the bug. Coverage of physical regimes, combined with reference-free physical oracles, predicts it well. A verifier built this way can reach the precision of reference-based test generators without needing a reference.
Three strands of evidence motivate the claim. Software testing has repeatedly found a gap between coverage and bug detection. Executing faulty code is not enough: the fault must also corrupt program state, and that corruption must reach an observable output (Aleti et al.). Fuzzers that reach more code do not reliably find more vulnerabilities (Liu et al.). Inputs that follow the same path differ greatly in whether they expose mutants (Qian et al.). Machine learning for physics found the parallel gap: a surrogate model can have low RMSE while violating conservation, boundary conditions, and the frequency spectrum (PDEBench). For generated scientific code specifically, InteractScience reports over 85% success on functional UI actions but visual-judge scientific scores often below 60, with perfect pass rates rarely above 16%. None of these works measures coverage against invariant preservation in generated simulation code. That measurement is our first deliverable.
Relation to Closest Work
Four papers come closest, and each plays a different role.
PHYFU supplies the core method. It fuzzes hand-written C++/CUDA physics engines (Taichi, Warp, Nimble, Brax) using physical-law oracles, such as the rule that distinct initial states must lead to distinct final states, and it found over 5,932 error-triggering inputs. It also constrains our design. Its oracles assume frictionless, deterministic, non-chaotic systems, and it tests engines rather than generated programs. We reuse the idea of physics as oracle. We add two things it lacks: oracles calibrated to tolerate the legitimate error of a correct numerical scheme, and inputs aimed at physical regimes rather than seeds near past errors.
Code2Worlds also targets generated simulation code. It repairs physical errors with a vision-language critic that watches one nominal rollout, and it acknowledges a trade-off between fidelity and latency. We replace that visual judgment with programmatic checks run across many regimes.
Code2Bench generates property-based tests, meaning tests that check general properties on many generated inputs using tools like Hypothesis, and keeps only those reaching 100% branch coverage on the ground-truth code. Even so, nearly 7% of submissions are near-misses that pass 98% of tests. Its rigor criterion is branch coverage, which is exactly the criterion we argue is uninformative for simulation loops. It also requires ground-truth code.
Reference-based verifier pipelines motivate the "no reference" setting. EvalPlus, HARDTESTGEN, and SAGA (Ma et al.) all derive expected outputs from reference solutions. HARDTESTGEN uses up to eight. All three work on stateless input-output tasks.
Research Program
The verifier has three parts.
Oracles. These are written once per physics family, not once per task. They come in three kinds. Conservation and dissipation checks cover energy, momentum, and mass, adapting the conservation and boundary metrics of PDEBench to program outputs. Metamorphic relations check how outputs should change under a transformed input: Galilean shifts, rescaling units, time reversal for reversible systems, and mass-independence of free fall. The last is the property that video models violate. Refinement-convergence checks handle a problem the other oracles cannot: a correct scheme also drifts, so a fixed tolerance would flag it. For invariant error $\epsilon(\Delta t)$ at timestep $\Delta t$, we require
$$\epsilon(\Delta t) \approx C\,\Delta t^{p}$$
with $p$ near the order of accuracy the program claims or the task specifies. A correct first-order scheme drifts, but its drift halves when $\Delta t$ halves. A sign error or missing term does not converge. This oracle is what lets us set tolerances without a reference solution.
Regime sampler. It draws physically valid inputs, such as positive masses and divergence-free initial velocity fields, from a grid over the dimensionless quantities above. It deliberately includes stiff, long-horizon, near-stability-limit, and near-zero-friction cases. This is how we escape the "homogenization trap" (Ma et al.), in which an LLM's own tests share its blind spots. The sampler's diversity comes from physics rather than from the model.
Regime-coverage metric. This is the fraction of regime cells in which the suite exercised at least one oracle. We report it next to branch coverage for every suite.
Baselines
The naive reference point is the status quo. It consists of the task's own example tests, or tests written by the same LLM, scored by pass/fail and branch coverage. This mirrors the thin suites criticized in EvalPlus and Sakharova et al..
The strongest reference-free competitor is property-based testing at the level of Code2Bench. It uses Hypothesis strategies inferred from argument types and keeps suites reaching 100% branch coverage. In the fair version, it receives the same physical oracles as our method and the same number of program executions. Only the input distribution differs: branch-coverage-guided type strategies versus our regime sampler. This baseline doubles as the key ablation, because it isolates whether regime-targeted inputs, rather than the oracles, carry the gain. A second ablation keeps the regime sampler and drops the refinement-convergence oracle, which tests whether tolerance calibration matters.
The visual-judge baseline follows Code2Worlds and InteractScience. A vision-language model judges rendered rollouts against the task description. It gets the same number of rendered scenarios as our method gets regime cells. InteractScience reports a Spearman correlation of 0.88 between its judge and human experts on static snapshots, so this is a serious competitor, not a straw man.
The upper-bound baseline is reference-based differential testing. A high-resolution trusted solver produces expected trajectories, and candidates are compared against them by trajectory RMSE. This is unavailable in our target setting. It shows how much is lost by going reference-free, and its RMSE-only version shows where RMSE and invariants disagree.
Beating the property-based baseline means higher verifier precision at matched recall, judged against held-out correctness labels, with a confidence interval excluding zero. For scale, HARDTESTGEN reports precision and recall gains of 11.22 and 11.03 points over its baselines on competitive programming, and TACO's precision falls as low as 17% on hard problems. No reported result was found on Lacuna for verifier precision on generated physics simulation code.
Why not just…?
Why not just run Hypothesis until branch coverage is 100%? This is the most serious threat. If Hypothesis inputs plus our oracles match our regime sampler, then the contribution is only the oracle library, which is still useful but narrower than claimed. We expect it to fall short for two reasons. Simulation bugs live inside unbranched loops, so full coverage is reached at trivial inputs. Code2Bench's 7% near-miss rate shows that branch-saturated suites still miss boundary cases. The matched-execution ablation above settles the question directly. We will report it whichever way it comes out.
Why not just use a vision-language model as judge? InteractScience states that its snapshot judge cannot assess temporal consistency. Generative vision models carry wrong physical priors, such as sub-Earth gravity (Thozhiyoor et al.). Code2Worlds reports a latency cost for its video critic. The judge may still catch gross errors cheaply, so we report its precision broken down by bug type. We expect it to do well on visibly wrong trajectories and to miss slow drift and stiff-regime blowups.
Why not just generate many more LLM tests? SAGA (Ma et al.) shows that volume yields diminishing returns because of homogenization. In physics code, random LLM inputs also produce invalid states, such as negative masses, which cause false failures. We include a volume-scaled LLM-test baseline at ten times the execution budget to show where its curve flattens.
Why not just compare against a reference solver? When a trusted solver exists, comparing against it is the right tool, and it is our upper bound. The point of this work is the case with no trusted solver: novel scenes and interactive demos. Even with a reference, RMSE misjudges chaotic systems that are correct but phase-shifted (PDEBench). We quantify how often that happens.
Proposed Experiments
We will build a labeled corpus of LLM-written simulation programs from three sources. The first is the physics demos of InteractScience, using the open-compass/InteractScience code, with the simulation core extracted. The second is solver-writing tasks for PDEBench systems: advection, Burgers', diffusion-reaction, and shallow water. The third is rigid-body and particle tasks in the style of Code4D. For each task we sample many candidate programs from several open and closed models. We label each candidate using a trusted high-resolution solver plus manual audit. These labels are used only for evaluation and are never shown to any verifier. We add mutation-injected variants of correct programs to obtain controlled bug classes.
The first decisive check is the gap measurement. Across suites, we regress bug detection on branch coverage and on regime coverage. Our claim predicts that branch coverage saturates near 100% with little predictive value while regime coverage carries the signal. The claim fails if the two predict detection equally well.
The second check is verifier quality: precision and recall of every baseline against the held-out labels, broken down by bug class and by regime.
The third check is downstream use. We run best-of-$k$ selection with each verifier and measure the true correctness of the chosen program, following HARDTESTGEN's rejection-sampling protocol. HARDTESTGEN reports LiveCodeBench pass@10 rising from 56.19% to 60.00%. We then run a small reinforcement learning fine-tune with our verifier as the reward, compared with a branch-coverage-gated property-based reward. We check whether policies trained on the latter learn to pass in benign regimes only, which would be reward hacking that the regime-coverage metric should expose.
References
- Defect Prediction Guided Search-Based Software Testing
- SBFT Tool Competition 2023 - Fuzzing Track
- Investigating Coverage Guided Fuzzing with Mutation Testing
- PDEBench: An Extensive Benchmark for Scientific Machine Learning
- InteractScience: Programmatic and Visually-Grounded Evaluation of Interactive Scientific Demonstration Code Generation
- PhyFu: Fuzzing Modern Physics Simulation Engines
- Code2Worlds: Empowering Coding LLMs for 4D World Generation
- Code2Bench: Scaling Source and Rigor for Dynamic Benchmark Construction
- Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation
- HARDTESTGEN: A High-Quality RL Verifier Generation Pipeline for LLM Algorithimic Coding
- Rethinking Verification for LLM Code Generation: From Generation to Testing
- Objects in Generated Videos Are Slower Than They Appear: Models Suffer Sub-Earth Gravity and Don’t Know Galileo’s Principle...for now
- Integrating Symbolic Execution into the Fine-Tuning of Code-Generating LLMs