Stiffness Bias in Force-Matched Interatomic Potentials

Neural networks trained by gradient descent learn some parts of a target faster than others. The best-known case is "spectral bias": smooth, low-frequency structure is fitted before sharp detail. This note argues that machine-learned interatomic potentials have a molecular version of the same problem, but with a twist. These are graph neural networks that map atom positions to an energy and are trained to match quantum-chemistry energies and forces. Here the components that get learned unevenly are the molecule's vibrational modes, its natural directions of motion. Some of these modes are stiff, such as bond stretches. Others are soft, such as torsions.

A short calculation shows that the standard force-matching loss favors stiff modes. Under thermal sampling, a given relative error in a soft mode's curvature produces a much smaller force error than the same relative error in a stiff mode. Gradient descent therefore spends its effort on bond stretches and under-trains the torsions and collective motions that govern conformational change. The energy loss does not have this imbalance, but it carries only one number per configuration. We propose a stiffness-whitened force loss that weights force residuals by an approximate inverse of the molecule's own Hessian (its matrix of second derivatives of energy with respect to atom positions). This loss is meant to keep the per-mode balance of the energy loss while retaining the information density of forces. We also test the claim directly by tracking per-mode convergence during training.

Core Claim

Consider a configuration near a local minimum. Along normal mode $k$, the energy is approximately $\tfrac12\kappa_k q_k^2$, where $\kappa_k$ is the mode's force constant and $q_k$ its displacement. Suppose a model misestimates the curvature by $\delta\kappa_k$. The force residual along that mode is then $\delta\kappa_k q_k$. Thermal sampling sets $\langle q_k^2\rangle \approx k_BT/\kappa_k$ by equipartition. The mode's expected squared-force contribution is therefore $(\delta\kappa_k/\kappa_k)^2\,k_BT\,\kappa_k$. In words, for the same relative error, stiff modes contribute in proportion to their stiffness.

The energy residual behaves differently. Its size is $\tfrac12\delta\kappa_k q_k^2 \approx \tfrac12(\delta\kappa_k/\kappa_k)k_BT$, which is the same across modes. Vibrational frequencies and conformational populations are what simulations actually need, and both depend on relative curvature error. Force matching thus weights the modes in a way that is mismatched to how the potential is later used.

We propose the loss

$$ \mathcal{L}_\alpha = \sum_{\text{configs}} \Delta\mathbf{F}^\top \big(|\tilde{H}| + \lambda I\big)^{-\alpha} \Delta\mathbf{F}, $$

where the terms are defined as follows. $\Delta\mathbf{F}$ is the mass-weighted force residual. $\tilde{H}$ is an approximate coordinate-space Hessian for that configuration, made positive by taking absolute eigenvalues. $\lambda$ is a floor that handles near-zero and rotational modes. $\alpha\in[0,1]$ interpolates between the two regimes: $\alpha=0$ is ordinary force matching, and $\alpha=1$ gives each harmonic mode equal weight in relative curvature error, matching the energy loss's balance.

The exponent $\alpha$ plays the role of the Sobolev order in Tuning Frequency Bias in Neural Network Training with Nonuniform Data. That paper shows that choosing the loss norm, with corrections for non-uniform sampling, can accelerate, damp, or invert the order in which frequency components converge. The difference here is that the "spectrum" is the molecule's physical vibrational spectrum rather than a Fourier basis. The non-uniform density is the Boltzmann distribution, whose spread along each mode depends on that mode's stiffness.

The prediction is concrete. Under Adam with $\alpha=0$, per-mode relative curvature error should fall in order of decreasing stiffness. With $\alpha$ near 1, the soft-mode error should fall faster. That gain should show up in torsion-sensitive extrapolation and in low-frequency vibrational accuracy, and it should not make the potential surface rougher.

Why This Is Not the Obvious Idea

The obvious combination of these two fields is to apply a spectrum-aware optimizer to equivariant networks. That has largely been done for static targets. How the Optimizer Shapes Learned Solutions in Equivariant Neural Networks trains EGNN and related models with Muon, an optimizer that orthogonalizes momentum updates. It reports higher-rank representations and gains on QM9 and ModelNet40, but it does not study force fields or training-time dynamics.

That work, and the curvature-preconditioning literature behind it, whitens the parameter-space geometry. Examples include Bonfanti et al. on Levenberg–Marquardt and Best McKay et al. on sketched natural gradients. We instead reweight the coordinate-space geometry of the target, using the physical Hessian of Phonax as a metric rather than as a label.

This distinction speaks directly to the tension in the file. Muon produces solutions that are sharper in parameter space. Liu et al. show that roughness in coordinate space, meaning force "cliffs" and spurious minima, is what breaks molecular dynamics. Our loss also deliberately does not amplify Fourier high frequencies. The "inverted" Sobolev regime would encourage exactly the roughness Liu et al. warn against. Instead, our loss rebalances across smooth physical modes.

Research Program

The first stage is a diagnosis. We will train MACE (Batatia et al., using the ACEsuit/mace code) and a PhysNet-style pairwise network. Every few hundred steps we will compute each model's autodiff Hessian at reference minima, diagonalize it in the DFT normal-mode basis, and record per-mode relative frequency error. This produces the per-mode convergence trajectory that the Muon study left unmeasured. It tests the stiffness-ordering prediction before any new loss is introduced. If force-trained models do not under-fit soft modes, the proposal stops here, and that negative result is itself worth reporting.

The second stage builds the reweighted loss. Computing $\tilde{H}$ once per training configuration is cheap for molecules in rMD17, 3BPA, and H₂CO; for example, 3BPA has 27 atoms, so its Hessian is 81 by 81. We will compute $\tilde H$ in two ways. One uses a fixed pretrained reference model, the MACE-OFF foundation model shipped with the MACE code. The other uses an exponential moving average of the model being trained, which avoids any external dependence. For larger molecules, as in MD22 or BSCT-SPICE, we will use a low-rank Lanczos approximation that captures the softest and stiffest modes, with $\lambda$ covering the rest. Neither variant needs DFT Hessian labels or third derivatives during backpropagation; the preconditioner is held fixed with a stopped gradient.

The third stage checks the regime where the harmonic argument breaks down. At high temperature and along dihedral cuts the potential is strongly anharmonic and $\tilde H$ can be indefinite. We will measure whether the gains persist there or reverse.

Baselines

The naive reference point is MACE trained exactly as in the reference implementation. It uses Adam, the default energy-to-force loss ratio, and the published architecture, run on each benchmark's standard splits. This anchors every comparison.

The strongest existing competitor is the same MACE model with the energy-to-force weight treated as a tuned hyperparameter. We will sweep that weight over the same number of trials and GPU-hours given to tuning $\alpha$ and $\lambda$. This matters because our own analysis says energy-heavy training already has the per-mode balance we want. For absolute numbers, MACE is reported to cut 3BPA 1200 K extrapolation error by roughly 30% relative to the best prior models and to reach its accuracy in about 30 minutes on one A100 (Batatia et al.). No reported absolute 3BPA or rMD17 MAE for MACE was found on Lacuna, so we will reproduce MACE ourselves and report our own numbers alongside the published relative gain. Beating this baseline means one of two outcomes. The first is a lower torsion-slice and 1200 K force error at equal or better 300 K error than any point on the tuned energy-weight frontier. The second is equal accuracy reached with fewer configurations.

The third baseline is MACE trained with Muon under the standard force loss, at matched step counts and learning-rate tuning. It isolates parameter-space whitening from coordinate-space whitening; the two are also run in combination.

On vibrational accuracy specifically, the reference points come from Käser et al.. They report harmonic-frequency accuracy of about 1 cm⁻¹ for PhysNet and about 0.2 cm⁻¹ for the kernel methods FCHL and RKHS+F on H₂CO. Closing that gap with a neural network would be a clear win.

The key ablation compares two preconditioner shapes. One is the full inverse-Hessian preconditioner. The other is an isotropic rescaling that keeps only the trace of $\tilde H$ and so has the same overall magnitude. If the isotropic version matches, the effect comes from loss scale, not mode balance, and the claim fails.

Why not just…?

Why not just upweight the energy term? This is the most serious threat, because energy residuals are already balanced across modes. Its weakness is information content. One scalar per configuration constrains a 3N-dimensional curvature far less than 3N force components. The data-efficiency results of Phonax make the general point that higher-derivative labels carry more information per configuration. The tuned energy-weight sweep settles this directly. If any energy weight matches $\alpha=1$ on both torsion extrapolation and in-distribution force error, we will report that the cheaper fix suffices.

Why not just use Muon or another curvature-aware optimizer? Those optimizers equalize learning across directions in weight space. The imbalance we describe is built into the loss itself: an ideal optimizer still minimizes a force objective whose optimum, under finite capacity and data, trades soft-mode accuracy for stiff-mode accuracy. The Muon-versus-$\alpha$ comparison, run alone and in combination, will show whether the two effects are separable.

Why not just train on DFT Hessians? Hessian labels fix curvature directly, but they scale steeply with system size and require third derivatives in training (Phonax). Our preconditioner reuses cheap model Hessians and leaves the labels unchanged. On H₂CO, where DFT Hessians are affordable, we will include Hessian-supervised training as an upper bound.

Why not just raise body order or add smoothing layers? Liu et al. report that MACE still shows force roughness on BSCT, and their differentiable neighbor lists target discontinuities rather than per-mode convergence. These fixes are orthogonal to ours, and we apply our loss on top of MACE.

Proposed Experiments

The decisive diagnostic is per-mode convergence on H₂CO (Set1 for interpolation, Set2 at 5000 K for extrapolation) and on rMD17. Finding that soft-mode relative error lags stiff-mode error under $\alpha=0$, and that the lag shrinks as $\alpha$ rises, confirms the mechanism. Not finding it falsifies the mechanism.

The decisive downstream test uses 3BPA. We will train at 300 K and evaluate force error on the 600 K and 1200 K slices and on the dihedral cuts, with the tuned energy-weight frontier as the comparison.

The safety check uses BSCT-SPICE Force Smoothness Deviation, which takes about 40 GPU-minutes per model (Liu et al.), and MD22 NVE stability runs. Together they confirm that rebalancing modes does not introduce cliffs. A model that improves torsion accuracy while raising Force Smoothness Deviation would count as a failure of the proposal, not a trade-off.

Finally, we will report $\alpha$-sweeps with learning curves at roughly $10^3$ and $10^4$ configurations. This is the data range where Fung et al. find graph networks become adequate, and where data efficiency should show most clearly.