Hidden-State Private Fine-Tuning of Catalyst Force Fields

Labs that design catalysts to suppress hydrogen evolution often hold their most valuable data as private quantum-chemistry calculations. Hydrogen evolution is the side reaction in which water is reduced to hydrogen gas instead of turning carbon dioxide or nitrate into useful products. A natural use of those calculations is to fine-tune a public machine-learned force field, a neural network that predicts the energy of an atomic configuration and the forces on its atoms, such as the models trained on the Open Catalyst 2025 solid–liquid data. Releasing the fine-tuned model, however, can leak the proprietary structures it was trained on. Differential privacy (DP), a formal guarantee that the released model barely changes if any one training example is removed, can prevent this. Standard DP accounting charges privacy at every gradient step, so the long, careful fine-tuning these models need becomes either unaffordable or too noisy to be useful.

This note proposes to fine-tune under the "hidden state" threat model: only the final weights are released, and every intermediate checkpoint stays private. Under that model, the theory of privacy amplification by iteration shows that privacy loss can stop growing with training length. The specific claim is that fine-tuning is the regime where this theory pays off. The weights only need to move a short distance from the pretrained model. Anchoring them to it with a proximal penalty and a projection onto a small ball supplies exactly the two conditions the convergent analyses require: contraction and a bounded domain. We expect a model that can be fine-tuned to convergence on private catalyst data, at a fixed and modest privacy budget, while keeping most of the accuracy gain that non-private fine-tuning brings on that domain.

Where the two directions meet

The electrocatalysis side supplies the application and the reason the data are sensitive. Selectivity against hydrogen evolution depends on fine atomic detail. Swapping one nitrogen for a carbon around a single nickel atom gives near-unity CO selectivity (Chen et al.), and alloying changes which hydrogen-transfer route dominates (Zhang et al.). These coordination motifs and alloy compositions are what a company would patent, and they are what a fine-tuned force field would memorize. OC25 provides 7.8 million DFT calculations over 1.5 million explicit-solvent environments, together with equivariant graph network baselines. It says nothing about training on data that cannot be shared.

The privacy side supplies the mechanism. Our work builds on three groups of papers.

The first group proves that loss converges when states are hidden, in increasingly general settings:

The second group supplies the analysis tools. These are shifted interpolation (Bok et al.), contraction in smooth ERM (Kifer et al.), and non-expansive fixed-point operators (Cyffers et al.).

The third group shows that the same machinery works on real networks. Koloskova et al. use noisy, clipped fine-tuning for certified unlearning on CIFAR-10 and an ImageNet-pretrained ResNet-18.

None of these papers evaluates on scientific regression, equivariant networks, or force fields. All empirical work is on vision, text, or synthetic losses.

Our role for these papers is specific:

  • We reuse the bounded-domain shifted-divergence analysis and the proximal-stabilization argument as the method.
  • We treat the bounded-domain requirement and the burn-in phase as constraints to engineer around.
  • We take the federated result as evidence that a proximal anchor is the right lever.

Core Claim

Let $\theta_0$ be the public pretrained weights. Let $g_t$ be the mean of per-example gradients, each clipped to norm $C$, on a minibatch of private configurations. Let $\Pi_B$ project onto the ball $B$ of radius $R$ around $\theta_0$. The update is

$$\theta_{t+1} = \Pi_B\big(\theta_t - \eta\,[\,g_t + \alpha(\theta_t - \theta_0)\,] + \xi_t\big), \qquad \xi_t \sim \mathcal{N}(0, \sigma^2 I).$$

Here $\eta$ is the step size, $\alpha$ the anchor strength, and $\sigma$ the noise scale. The projection meets the bounded-domain condition of the non-convex analysis by construction.

The proximal term adds curvature $\alpha$. Wherever $\alpha$ exceeds the most negative curvature of the fine-tuning loss inside $B$, each step contracts distances between neighboring trajectories by roughly $(1-\eta\alpha)$. This mirrors the role of the proximal term in FedProx.

In these analyses the converged privacy bound grows with the domain diameter and shrinks with the contraction rate. Fine-tuning is therefore the regime where both quantities are favorable: $R$ can be small because fine-tuned weights stay close to pretrained ones, and $\alpha$ can be meaningful without losing accuracy.

The new contributions are three:

  • a privacy accountant for this anchored, projected update that is explicit in $R$, $\alpha$, $\sigma$, $C$ and the batch scheme;
  • an empirical estimate of local curvature inside $B$, which checks when the contraction regime actually holds for an equivariant force field;
  • the first evaluation of hidden-state private training on a scientific regression task where errors are measured in tens of meV.

Research Program

Accountant. We will specialize the shifted Rényi divergence analysis to the anchored and projected map, giving a closed-form plateau $\varepsilon_\infty(R,\alpha,\sigma,C,n)$ and the burn-in length. We will also state honestly where the bound needs smoothness that clipping breaks. In that case we fall back on the Hölder-continuous version, which does not need Lipschitz gradients.

Curvature check. Fine-tuning only part of the network keeps the analysis and compute tractable. We will fine-tune the last interaction blocks and the readout of an OC25-pretrained eSEN model, with the earlier layers frozen. Along the training trajectory we will estimate the smallest Hessian eigenvalue within $B$ using Lanczos iterations. This tells us whether the chosen $\alpha$ actually puts the loss in the contractive regime, or whether we rely only on the weaker non-convex bound.

Private data. We need a "private" domain without inventing data. We will hold out a chemically coherent slice of OC25 and treat it as proprietary. The primary slice is all environments whose bulk contains Cu and/or Ag, matching the Cu–Ag systems optimized in Grega et al.. A second slice holds out Ni-containing systems. We re-pretrain the backbone on the remainder of OC25, so that $\theta_0$ has truly never seen the private slice.

Selectivity check. As a secondary, application-facing check, the slice may contain enough hydrogen-adsorbate configurations. If so, we will also measure whether the private model preserves the ranking of candidate surfaces by hydrogen binding energy, the standard descriptor of how strongly a surface will evolve hydrogen. We have not confirmed that OC25 contains these configurations. If it does not, the evaluation stays at energy, force and pseudo-solvation error.

Baselines

Do nothing. The naive reference point is the pretrained model applied zero-shot to the private slice. It has perfect privacy at no cost and sets the floor that any private method must clear to be worth running.

Non-private ceiling. The ceiling is ordinary fine-tuning on the private slice with the same architecture, frozen layers, steps and tuning budget. The full-data OC25 baselines give a sense of scale. eSEN-S-cons reports 0.105 eV energy MAE and 0.015 eV/Å force MAE, and eSEN-M-d reports 0.040 eV pseudo-solvation MAE (OC25). Those numbers are for a different split, so we will recompute the ceiling on ours. We found no reported result on Lacuna for any differentially private training of interatomic potentials on OC25.

Strongest existing private method. This is DP-SGD with the same clipping and noise, accounted by standard Rényi composition. Under composition the budget grows with steps, so we run it two ways. One version is trained at the same total steps and reports its larger $\varepsilon$. The other is stopped at whatever step count meets the target $\varepsilon$ and reports its accuracy. Both get the same tuning budget per $\varepsilon$ as our method.

Convex-head competitor. This is a private linear or ridge readout on frozen pretrained embeddings, fitted once with a one-shot private regression in the style of aggressive-clipping private linear regression. We also run output perturbation of the non-private fine-tune, which Koloskova et al. found clearly worse than iterative noisy fine-tuning on vision.

Ablations. The key ingredient is isolated by three variants of our method:

  • $\alpha=0$ with projection kept;
  • no projection with $\alpha$ kept;
  • neither.

What counts as winning. Beating the baselines means two things at matched $\varepsilon$ of 1, 4 and 8 (with $\delta=1/n$). First, our method must recover a substantially larger fraction of the zero-shot-to-ceiling gap in force MAE on the private slice than both composition-accounted DP-SGD and the private convex head. Second, membership-inference success must be no higher than for those baselines.

Why not just…?

Why not just use standard composition with few steps? If good accuracy were reachable in very few steps, composition would cost little and hidden-state accounting would add nothing. This is a real threat because of burn-in: below the burn-in length the two accountants agree, as the non-convex analysis notes. The "stopped at target $\varepsilon$" composition baseline settles it directly. We will also report the step count at which the two accountants diverge. If non-private fine-tuning converges well before that step count, we will report that the proposal does not help in this regime.

Why not just fit a private linear head on frozen features? A strongly convex head allows clean one-shot private regression, and it may well match us when the private domain is close to the pretraining distribution. Our bet is that a held-out element family shifts the representations enough that deeper layers must adapt. The convex-head baseline, run at matched $\varepsilon$, decides this. Losing to it on the Cu/Ag slice but beating it on the Ni slice, or vice versa, would itself show which domain shifts need deep private adaptation.

Why not just apply this to the autonomous-lab loop of Grega et al.? That loop works on about 90 electrodes from 30 compositions. A Gaussian-process posterior over so few points is not a long noisy iteration. Any meaningful DP guarantee on 30 compositions would swamp the signal, and the small gain of the physics-informed model over a plain GP reported in Grega et al. leaves no room to absorb that noise. The non-smoothness concern of optimizing over physical parameters, such as restructuring under bias (Girotto et al.), also does not arise here. We optimize network weights, which are smooth in the parameters inside $B$.

Why not just share data under contract or train inside a secure enclave? These measures protect the data while training happens, but not the released model, which can still be probed for memorized structures. Our membership-inference evaluation measures exactly this residual leakage for the non-private ceiling.

Proposed Experiments

The first decisive check is the privacy curve. We plot $\varepsilon$ against step count for the anchored-projected accountant and for composition at the noise level actually used, over training long enough to pass burn-in. The anchored accountant should flatten while composition keeps rising, and the plateau should move with $R$ and $\alpha$ as the bound predicts. The curvature measurements tell us whether the plateau rests on the contraction argument or only on the bounded-domain argument.

The second check is utility at matched privacy. On both held-out slices we report energy MAE, force MAE and pseudo-solvation MAE for every baseline and ablation at three $\varepsilon$ values. Each configuration uses three seeds and an equal tuning budget. Where the data allow, we also report Spearman correlation of hydrogen binding energies with DFT. The $\alpha=0$ ablation is the critical comparison. If removing the anchor costs little privacy at fixed accuracy, the claimed mechanism is wrong even if the method works.

The third check is an empirical attack. We run loss-based membership inference against private-slice configurations, comparing our model, the non-private ceiling, and composition-accounted DP-SGD. The analogous attacks in the federated study stabilized rather than climbing toward 100% as training continued. We expect the same qualitative flattening under long anchored fine-tuning.

As a cheap extension, the same accountant directly gives certified removal of a withdrawn collaborator's structures. This is done by continued anchored noisy training on the retained data, following Koloskova et al.. We will report steps-to-recover against retraining from $\theta_0$.