Is It the Potential or the Plumbing? Differential Testing Across Bindings of Machine-Learned Force Fields

Machine-learned interatomic potentials are neural networks that predict the energies and forces on atoms. They let materials scientists run simulations that would be far too expensive with quantum-mechanical calculations, whose cost grows with the cube of system size. The same trained model, though, usually reaches a simulation through several different code paths. It might be called from Python, exported into a C++ molecular dynamics engine such as LAMMPS, accelerated with custom CUDA kernels, or split across GPUs by a distributed library. Each path crosses a boundary between programming languages, and software engineering research has shown that such boundaries are where bugs concentrate.

We propose to exploit a property that ordinary software lacks. Physics provides a free test oracle, meaning an independent way to decide whether an output is correct. Every execution path of one potential must return the same energies and forces on the same atoms. Those outputs must also be unchanged under rotation, permutation of identical atoms, and repetition of the periodic cell, and forces must equal the negative gradient of the energy. Checking these relations across bindings can expose cross-language defects that never crash and so never reach an issue tracker. It can also answer a question the field currently cannot: when a simulation blows up, is the model at fault, or the glue code?

Core Claim

The core claim is empirical and has two parts, both falsifiable. First, systematic cross-binding differential testing will surface confirmed, previously unreported defects at the language boundaries of widely used open-source potential stacks. Differential testing here means running identical inputs through each execution path and comparing outputs against a calibrated numerical noise floor, together with physics-invariance checks. The dominant class of these defects will be silent semantic errors, not the memory errors that dominate in deep learning frameworks. Examples include unit and sign conventions for stress, handling of periodic images when the cutoff approaches half the cell width, precision downcasts, and inconsistent treatment of atoms at partition boundaries.

Second, this implies a specific bias in the existing empirical record of multi-language bugs. That record is mined from public issue trackers, so it over-represents failures that are loud enough for someone to file.

The second claim is where the dataset-bias direction enters. Understanding Bugs in Multi-Language Deep Learning Frameworks mined 1,497 bugs from PyTorch, TensorFlow and MXNet. It found that over 92% of multi-language bugs sit at the Python–C/C++ boundary and that memory bugs are the most communication-intensive. Reflections on Software Failure Analysis found that 94% of failure studies examine only open-source software and only 19% reuse an existing bug taxonomy. Both results describe what can be observed passively in mature, well-staffed public projects.

Scientific stacks are written largely by domain scientists and validated by eyeballing trajectories. We expect them to hide a different distribution of defects, and the only way to see it is to generate failures actively rather than collect them. The cost direction supplies the stakes and the regime. These stacks exist to avoid the cubic cost of first-principles calculations. A silent binding error therefore quietly invalidates simulations that are, by construction, never cross-checked against the expensive reference.

Positioning Against Prior Work

The work that motivates this proposal falls into three groups. The first measures boundary friction in general software. Exploring Multi-Programming-Language Commits and Their Impacts on Software Quality shows that commits spanning languages are scattered across architectural tiers and correlate with defects. Understanding Resolution of Multi-Language Bugs shows that friction concentrates at the interface of exactly two languages. We take that second finding as a design constraint: our harness targets one boundary at a time, Python-to-TorchScript/C++ and Python-to-partitioned-GPU. None of these studies generate failures; they mine them.

The second group consists of tools that analyze boundaries directly. Automated Detection of Inter-Language Design Smells introduced CPSMELL, a static detector of seven design smells in pybind11, ctypes and the Python/C API, with 98.17% precision on deep learning frameworks. Toward efficient interactions between Python and native libraries introduced PieProf, a profiler that reconstructs call paths across the Python/native boundary to find performance waste in NumPy and Scikit-learn. CPSMELL detects structural smells rather than wrong outputs, and PieProf detects slowness rather than incorrectness. Neither has an oracle for numerical correctness, which is exactly what physics provides.

The third group is the atomistic stack we study. MACE is a fast equivariant potential whose ecosystem spans PyTorch, JAX, CUDA bindings and interfaces to the Atomic Simulation Environment (ASE, a Python toolkit for atomistic simulation). DistMLIP partitions the atom graph across GPUs to scale MACE, CHGNet, TensorNet and eSEN toward a million atoms. DistMLIP notes that some equivariant features must be computed on a single GPU "for numerical stability." That detail is direct evidence that execution paths are not numerically interchangeable for free. Both papers evaluate speed and accuracy against quantum-mechanical labels, not agreement across their own bindings.

Two papers constrain rather than motivate. Energy & Force Regression on DFT Trajectories is Not Enough and OC20 attribute unstable or inaccurate simulations to the model and its training data. We do not dispute this. Our point is that their diagnosis currently has no control for plumbing errors, and our harness supplies one.

Finally, CrossPL is the closest pairing of language boundaries with open-source data effects. It shows that GPT-4o reaches only 19.54% Pass@1 on Python-calls-C foreign-function tasks built around the GNU Scientific Library. CrossPL, however, measures greenfield code generation on isolated wrappers. We instead diagnose defects in existing, deployed scientific bindings. Its result enters our work only as a reason to expect a language-model reviewer to be a weak baseline.

Research Program

We will build an open harness that loads one trained potential, for example MACE-MP-0b-small, CHGNet, TensorNet and eSEN as used in DistMLIP. It will evaluate that potential through every available execution path: the native PyTorch model through ASE, the exported model inside LAMMPS, the CUDA-accelerated equivariant kernels, and DistMLIP with varying numbers of partitions. It will compare energies, per-atom forces and cell stresses on the same configurations.

The comparison tolerance is calibrated rather than guessed. We will estimate the noise floor by running each path repeatedly and running the reference path in 64-bit precision. A discrepancy counts as a finding only when it exceeds that floor by a margin fixed before testing begins.

On top of cross-binding agreement, we add metamorphic relations. A metamorphic relation is a rule stating how an output must change, or stay the same, when the input is transformed in a known way. Ours are rotation and translation invariance, permutation of same-species atoms, extensivity when the periodic cell is replicated, force as the negative finite-difference gradient of energy, and invariance of the result to how atoms are partitioned across GPUs. In a microcanonical simulation, where total energy should be conserved, we also measure energy drift.

Configurations will be drawn from the regimes where boundary code is most stressed and least exercised. These include the out-of-domain splits of OC20, the high-temperature 3BPA extrapolation sets and reactive acetylacetone frames from the MACE benchmarks, and adversarially constructed geometries. The adversarial cases are small or strongly skewed (triclinic) cells, cutoffs near half the box width, mixed species near partition walls, and near-collision pairs. None of this requires new quantum-mechanical calculations, because the reference is the model itself. The whole study therefore costs GPU-hours of inference, not supercomputer allocations.

Each confirmed discrepancy is reduced to a minimal reproducing input and traced to the responsible boundary. It is then classified using the existing multi-language taxonomy from the deep learning framework study, extended only where necessary, in response to the taxonomy-fragmentation critique. It is reported to maintainers.

In parallel, we mine the public issue trackers of the same codebases and classify issues with the same taxonomy. This yields two defect distributions for the same software: one passively observed and one actively generated. The gap between them is our measurement of issue-tracker bias.

Baselines

The naive reference point is the status quo: the potentials' own continuous-integration test suites plus whatever defects appear in public issue trackers. We run each repository's shipped tests at the same pinned versions and count which of our confirmed defects they would have caught. We count mined issues as the "found passively" baseline. No reported result was found on Lacuna for the defect-detection rate of these suites. Beating this baseline means confirmed boundary defects that the suites pass and the trackers do not list.

The strongest existing correctness-oriented competitor is energy-conservation monitoring in microcanonical molecular dynamics. It is the sanity check practitioners already run, and it needs no second binding. It is also the most direct challenger to our key ingredient. We run it on the same configurations with the same GPU budget as the full harness. It is therefore also our main ablation: full harness versus drift-only, then versus single-path metamorphic checks without cross-binding comparison. This isolates whether comparing bindings, rather than physics checks alone, is what finds defects. We count the harness as winning if a substantial share of confirmed defects, which we will fix before looking at results, is caught only by cross-binding or partition-invariance checks.

The strongest existing boundary-analysis tool is CPSMELL from Automated Detection of Inter-Language Design Smells. We run it unmodified on the Python/C++ binding code of each stack and, where its parsers apply, on the export code. We then measure overlap between its smell locations and our defect locations. Its reported figure is 98.17% precision for smell detection on deep learning frameworks. That measures smell precision, not defect recall, so no reported result was found on Lacuna for CPSMELL on atomistic codes or on defect-finding. Beating it means our defects mostly lie outside flagged smell sites.

We also include a language-model code reviewer. A frontier model is given each binding source file and asked to flag correctness defects, with the same number of queries per file as we spend on triage. The strongest related number in the file is 19.54% Pass@1 for GPT-4o on CrossPL-FFI, from CrossPL. That is a generation task, not a review task, so we treat it only as a prior.

Why not just…?

Why not just watch energy drift in microcanonical runs, which every practitioner already does? This is the real threat. If drift catches nearly everything, the elaborate harness is unnecessary. We believe it will not, for two reasons. First, a constant scaling error in stress, or a consistent error in how periodic images are counted, can conserve energy perfectly while giving wrong pressures or densities. Second, drift conflates model roughness out of distribution, the failure documented in Energy & Force Regression on DFT Trajectories is Not Enough, with plumbing errors, so it cannot attribute blame. The drift-only ablation settles this directly. If drift finds as many confirmed defects as the full harness, we will report that and narrow the claim to attribution.

Why not just run a static smell detector like CPSMELL? Smells are structural patterns that make defects more likely, such as unused native entities or long lambdas. A detector for them has no notion of whether the numbers crossing a boundary are correct. A unit-convention or partitioning error can live in clean, smell-free code. The overlap measurement described under Baselines shows how much the two approaches coincide. If the overlap is high, CPSMELL is the cheaper tool, and we would say so.

Why not just rewrite everything in one language, for example the JAX version of MACE, so there is no boundary? Production-scale simulation still runs inside compiled engines like LAMMPS. DistMLIP matched LAMMPS's custom CUDA kernels only by adding native communication code, so boundaries persist in practice. A single-language path is also exactly the kind of reference implementation our differential test needs, so it complements this work rather than replacing it.

Why not just ask a code model to review the bindings? CrossPL shows that even the best model writes correct foreign-function glue under a fifth of the time. Reviewing for silent numerical conventions requires knowing, for instance, what sign and units LAMMPS expects for virial stress. That knowledge is exactly what these models lack. We include the reviewer as a baseline rather than assuming it fails.

Proposed Experiments and Decisive Checks

The first decisive check is retrospective and has ground truth. We check out historical releases of each stack immediately before commits that fixed boundary-related bugs, run the harness blind, and measure what fraction of those later-fixed bugs it would have flagged. We then compare that fraction to the drift-only and test-suite baselines. If the harness recovers few known fixed bugs, the approach fails regardless of anything else.

The second check is prospective. On current releases, we report the number of distinct defects that maintainers confirm or that minimal cases reproduce. We also report the false-alarm rate, meaning discrepancies above tolerance later attributed to legitimate nondeterminism, and GPU-hours per confirmed defect.

The third check tests the bias hypothesis. We compare the category distribution of actively found defects with that of mined issues for the same repositories, and with the deep learning framework distribution. The hypothesis is falsified if memory and crash defects dominate both distributions and silent semantic defects are rare in both. In that case the existing open-source bug record would transfer to scientific stacks better than we expect, which would itself be worth reporting.

The fourth check is about attribution. For simulations that go unstable on OC20 out-of-domain and 3BPA high-temperature inputs, we rerun the same trajectory through the reference binding in 64-bit precision. We then report what fraction of instabilities persist, meaning they are model failures, versus vanish, meaning they are plumbing failures. This number is currently unknown for any of these stacks and bears directly on how the literature interprets out-of-domain failures.