Which Reference Calculations Set the Melting Point? Data Deletion Audits for Fitted Interatomic Potentials

Simulations of materials and molecules usually replace quantum mechanics with a simplified interaction potential. This is a cheap formula, such as the Stillinger-Weber potential for silicon, with a handful of parameters fitted to a database of quantum reference calculations. Practitioners then run long simulations with the fitted formula and report properties like melting temperatures, liquid structure, or phase diagrams. Nobody currently checks how much those reported properties depend on individual entries in the fitting database. This note proposes a leave-one-out audit that answers that question. Each reference configuration, or each correlated group of configurations, is deleted in turn, the potential is refit, and the audit measures how far the simulated physical property moves. We then test whether the largest such shift is larger than chance would produce.

Two facts make this feasible and interesting. First, a simplified potential has so few parameters that exact refitting after a deletion takes seconds. Leave-one-out is expensive for neural networks, but here it is nearly free. The real cost is re-running the simulation, and we avoid most of it by pushing each refit's parameter shift through a once-computed sensitivity of the observable to the parameters. Second, these potentials are known to be "sloppy": a few parameter combinations are tightly pinned by the data, while many others can drift over orders of magnitude without changing the fit (Kurniawan et al.). Our hypothesis is that the configurations which most move simulated observables are not the ones with large residuals, and not the ones that most move the training fit. Instead, they are the few configurations that happen to be the only thing anchoring a loosely determined parameter direction that a downstream observable cares about.

Core Claim

For a fitted potential with parameters $\theta$, let $\theta_{(-i)}$ be the refit after deleting configuration (or group) $i$, and let $O$ be a simulated observable such as melting temperature. The quantity of interest is the observable influence

$$\Delta O_{(-i)} \approx g_O^\top \left(\theta_{(-i)} - \hat\theta\right),$$

where $\hat\theta$ is the full-data fit and $g_O$ is the gradient of the simulated observable with respect to the parameters. The parameter shift $\theta_{(-i)} - \hat\theta$ is computed by exact refitting, not by an approximation. The gradient $g_O$ is estimated once, either by thermodynamic reweighting of a single trajectory or by finite differences over the few parameters.

The claim is empirical and falsifiable. On standard silicon and argon fitting databases, the ranking of configurations by $|\Delta O_{(-i)}|$ will differ substantially from rankings by residual size, by training-loss influence, and by parameter-space curvature. A small number of deletions will shift observables by more than the uncertainties that published simulations report. We also want to know whether a deletion's effect is statistically unusual, not just large. For that we adapt the extreme-value test of Testing Most Influential Sets, which gives p-values for the maximum influence in linear regression. We extend it to this nonlinear, correlated setting with a parametric-bootstrap null.

Relation to Prior Work

The closest work studies sensitivity in the wrong space. Kurniawan et al. map parametric uncertainty of Lennard-Jones, Morse, and Stillinger-Weber potentials using MCMC, profile likelihoods, and information geometry. They show sloppiness and fix some of it by reparameterization. They perturb parameters, never the data, and never propagate to simulated observables. Their sloppiness result is the motivation for our hypothesis, and their MCMC samples also supply one of our baselines.

KLIFF supplies the method substrate: a fitting pipeline for Stillinger-Weber and neural potentials, with Fisher-information-guided optimization and dropout-based uncertainty at inference time. We reuse its fitting pipeline. We add something it lacks, namely attribution of fitted behavior to specific training configurations.

Forces are not Enough motivates targeting simulated observables rather than force error. It shows that low force error does not predict stable or physically faithful simulation. However, it evaluates fixed train/test splits and never traces failures back to training data.

The leave-one-out literature constrains the design. Testing Most Influential Sets and Konrad & Kuschnig give exact deletion formulas and extreme-value tests, but only for linear least squares. Their finding that first-order approximations underestimate group effects is why we refit exactly. Structured-data work shows that deleting a single correlated point biases leave-one-out estimates (Distributional bias compromises leave-one-out cross-validation; Automatic cross-validation in structured models). Reference configurations drawn from the same trajectory are strongly correlated, so our primary unit of deletion is a group (a trajectory segment or a structure family), with single configurations as a secondary analysis. The classical-ML leave-one-out shortcuts (Leave-One-Out Support Vector Machines; Quick Sensitivity Analysis for Incremental Data Modification) target training loss in convex models. They are not needed here because exact refits are cheap, and they do not address downstream simulation.

Research Program

We build the audit on three testbeds. The first is Stillinger-Weber silicon fitted in KLIFF to a quantum reference database of bulk, defect, and liquid configurations. Its observables are melting temperature, liquid density and coordination number at fixed temperature and pressure, radial distribution function, and diffusivity. The second is Lennard-Jones and Morse argon, a low-dimensional control where the audit can be checked exhaustively. The third, a stretch target, is the supercooled-silicon liquid-liquid transition. For the original Stillinger-Weber parameters, Liquid–liquid critical point in supercooled silicon reports $T_c \approx 1120 \pm 12$ K and $P_c \approx -0.60 \pm 0.15$ GPa. Those error bars give a natural yardstick: does deleting any single reference group move the predicted critical point by more than the statistical uncertainty that the paper reports?

The analysis has three parts. First, we compute exact deletion refits for every group, warm-started from $\hat\theta$. We then decompose each parameter shift onto the eigenvectors of the fit's Gauss-Newton matrix, to test whether large observable influences come from shifts along poorly determined directions. Second, we estimate $g_O$ for each observable. For structural and thermodynamic averages we reweight a single trajectory; for melting temperature and diffusivity, where reweighting is unreliable, we use central finite differences over the roughly seven parameters. Third, we build a significance test. We generate synthetic reference datasets from $\hat\theta$ with noise matched to the fit residuals and refit them. We then run the full deletion audit on each synthetic dataset and fit an extreme-value distribution to the maximum $|\Delta O_{(-i)}|$. This generalizes the Fréchet and Gumbel analysis of Konrad and Kuschnig without assuming linearity.

Baselines

The naive reference point is residual-based pruning. The practitioner's habit is to flag the configurations with the largest energy or force residuals as suspicious. We rank groups by residual norm using the same full-data fit and compare that ranking with the true observable-influence ranking. Beating it means a clearly higher rank correlation, and substantially higher overlap in the top ten groups, against brute-force ground truth.

The strongest existing method for approximate data deletion is the first-order influence function: the inverse Hessian of the fitting loss times the per-configuration gradient, projected onto $g_O$. It is the standard tool, and here it is cheap. We run it with the same $g_O$ and with damping tuned on a held-out set of groups. The damping is tuned under the same budget of brute-force checks we allow ourselves, so the only difference from our method is exact refitting versus linearization. No reported result was found on Lacuna for influence-function accuracy on interatomic potentials. The only nearby figures come from linear regression in Testing Most Influential Sets, for example a 17-point set flagged at $p = 0.019$ on the Law School data. That figure is context, not a target.

The second strong competitor is posterior importance-sampling leave-one-out. Using the MCMC parameter samples of the kind produced by Kurniawan et al., each sample is reweighted by the inverse likelihood of the deleted group, and the observable is averaged over the reweighted samples. We run it with the same total number of sampler likelihood evaluations as our refits. No reported result was found on Lacuna for this estimator on interatomic potentials.

Parameter-space curvature from the Fisher information matrix, as used in KLIFF, serves as a non-attributive reference. We rank groups by their contribution to the information along $g_O$.

The key ablation removes the observable projection. We rank groups by the norm of their parameter shift, or by their effect on held-out force error, instead of by $|\Delta O_{(-i)}|$. If that ranking matches the observable ranking, the core claim fails.

The approximations are judged against ground truth: full refit plus re-simulation for all groups on argon, and for the top thirty plus thirty random groups on silicon. Our method beats a baseline if its Spearman correlation with ground-truth $\Delta O$ is higher and its top-ten recall is higher, with the gaps outside bootstrap intervals.

Why not just…?

Why not just look at the Fisher information or Hessian of the fit? It says which parameter directions are poorly determined, but not which data points determine them, and the fitting practitioner can act only on data. Sloppiness also makes the Hessian badly conditioned (Kurniawan et al.), so curvature rankings are unstable along exactly the directions we care about. The curvature baseline above measures how much attribution it recovers anyway.

Why not just use first-order influence functions, since they are cheap? This is the real threat. If the fitting loss is close to quadratic around $\hat\theta$, they will match exact refits, and our contribution shrinks to the observable projection and the test. We expect them to fail for groups that pin sloppy directions, where deletion moves the fit far along a curved valley. Group deletion is also where Testing Most Influential Sets documents underestimation. The head-to-head comparison against brute force settles it. If the influence function wins, we will report that and recommend it, since it is cheaper still.

Why not just use ensemble or dropout uncertainty during simulation? Model disagreement at inference time flags states that the potential is unsure about. It cannot name the training configuration responsible, and Forces are not Enough finds that such flags do not prevent failure without new data. Our output tells the practitioner which reference groups to recompute or augment.

Why not just run k-fold cross-validation? Averaged folds measure typical predictive error and dilute the effect of rare, isolated anchors. Structured-data work shows that fold choice matters under correlation (Automatic cross-validation in structured models). We include random-fold variance of $O$ as a reference and expect it to be small next to the maximum group influence. If it is not, influential groups are not special, and we would report that.

Decisive Checks

The first check is whether any single group deletion on silicon moves the melting temperature, or the argon liquid density, by more than its simulation error bar, with an extreme-value p-value below 0.05. This must hold against the bootstrap null and not only in absolute terms. The second check is whether the top observable-influence groups have shifts that load mainly on the smallest-eigenvalue directions of the Gauss-Newton matrix, and whether residual and loss-influence rankings miss them. The third check is actionability. We recompute the flagged groups with tighter quantum settings, or add neighboring configurations, and refit. The observable's spread across deletions should shrink, while held-out force error stays within 5% of its original value. On the liquid-liquid transition, the stretch result is either that some deletion moves $T_c$ beyond ±12 K or that none does. Either outcome is informative about how robust published phase-diagram claims are. The argon testbed is exhaustive: every group is refit and re-simulated there, so the approximations are checked against ground truth for all groups rather than a sample.