Teaching SCF Warm Starts Where the Charge Goes

Quantum chemistry codes find a molecule's electronic ground state with an iterative loop called the self-consistent field (SCF) procedure. The loop guesses an electron density, builds the potential that density creates, solves for a new density, and repeats until the two agree. A neural network that predicts a good starting density can cut the number of loop iterations. Because the classical solver still runs to full convergence, the final answer is as exact as ordinary density functional theory (DFT). The best transferable version of this idea so far predicts electron-density coefficients with an equivariant graph network, meaning a network whose outputs rotate correctly when the molecule rotates. It removes about a third of SCF iterations, while the authors estimate that a perfect density would remove about 60% (Towards a Transferable Acceleration Method for Density Functional Theory).

This note argues that most of that gap has a specific cause. The SCF loop converges slowly on long-wavelength charge errors: too much charge on one end of a molecule, or a wrong dipole. Those are exactly the errors a local message-passing network cannot see, and a per-coefficient regression loss does not prioritize them. We propose three changes to fix this. First, a linear-cost global attention stage, so every atom can learn about distant charge. Second, a Coulomb-metric loss that penalizes errors by their electrostatic energy. Third, a shared-weight ("Siamese") pair loss that trains the model to predict how the density in one region changes when a distant fragment is removed. The test is whether iteration counts on SCFbench move from the reported ~66% toward the ~40% ceiling, including on molecules three times larger than those seen in training.

Core Claim

The claim is that warm-start quality for SCF depends on the error in the slow modes of the solver, not on average coefficient error. The dominant slow mode in finite molecules is long-range charge redistribution, which reflects polarization and charge transfer across the molecule.

A predicted density that is locally accurate but has the wrong global charge distribution produces a large error in the electrostatic (Hartree) potential. The SCF loop then spends many iterations damping that error away. If this claim is right, three things should hold. Networks that let distant atoms communicate should beat purely local ones on iteration count by more than they beat them on coefficient error. A loss that weighs errors by electrostatic energy should reduce iterations even at fixed architecture. The benefit of both should grow with molecule size, because the room for long-range error grows with size.

Two pieces of prior work motivate this. The first is the transferable-acceleration study cited above. It shows that density targets transfer from 20-atom training molecules to 26–60-atom test molecules at a nearly constant relative iteration count, while Hamiltonian targets collapse to 179%. That makes density the right target, but it leaves a flat, unexplained gap to the ceiling. The second is AllScAIP (A recipe for scalable attention-based ML potentials), which adds all-to-all atom attention to an interatomic potential. It reports roughly 90% lower error than local models such as eSEN and UMA when molecules are artificially stretched. That is direct evidence that local cutoffs miss long-range electronic effects. AllScAIP, however, predicts energies and forces rather than densities, and its attention cost grows quadratically with atom count. MatRIS (MatRIS) shows that linear-cost separable attention can match equivariant accuracy on Matbench-Discovery at up to 13× lower training cost. Its outputs are invariant scalars, so it cannot by itself produce the direction-dependent density coefficients a warm start needs. We reuse its lesson that global mixing can be done cheaply on invariant channels, and we attach it to an equivariant density head.

Research Program

The model starts from the density-predicting equivariant network of the transferable-acceleration study (NequIP-L), which is the strongest reported density warm start. After its local layers, we insert a global stage based on linear attention. Linear attention replaces the softmax over all atom pairs with a feature map $\phi$, so that attention can be computed as $\phi(Q)(\phi(K)^\top V)$ in time linear in atom count (Efficient Attention: Attention with Linear Complexities).

The queries, keys and values are built only from rotation-invariant scalar channels of each atom: learned charge-like and polarizability-like features plus distance-encoded positional terms. The attention output then modulates the equivariant (vector and tensor) channels multiplicatively. This keeps the whole model exactly rotation-equivariant, which we verify numerically rather than assume. It also avoids the approximate, emergent equivariance that AllScAIP relies on.

We expect linear attention to be adequate here even though Why Softmax Attention Outperforms Linear Attention shows it is weak at sharp one-to-one retrieval. Electrostatic influence is smooth and nearly low-rank at long range, which is the regime where low-rank kernels are accurate. That paper constrains the design: we do not use linear attention for the short-range bonding structure, which stays with the local layers.

The first new loss replaces the per-coefficient error with the Coulomb metric. If $c$ and $c^*$ are the predicted and reference auxiliary-basis coefficient vectors, the loss is

$$\mathcal{L}_J = (c - c^*)^\top J\,(c - c^*), \qquad J_{kl} = \iint \frac{\chi_k(\mathbf r)\,\chi_l(\mathbf r')}{|\mathbf r - \mathbf r'|}\,d\mathbf r\,d\mathbf r',$$

where $\chi_k$ are the auxiliary basis functions. The loss equals the electrostatic self-energy of the density error. It therefore punishes misplaced net charge and dipole errors far more than equally sized errors that cancel locally. This is the same quantity that drives the potential error the SCF loop must remove. We also add a soft penalty on total-charge mismatch, because a density that does not integrate to the right electron count cannot be converted into a valid starting potential.

The second new loss is the Siamese response term. For a training molecule $M$, we cut a distant fragment, cap the broken bonds with hydrogen, and compute the DFT density of the remaining molecule $M'$. The same network, with shared weights, is run on both. On the atoms that remain in both, the loss compares the predicted change in their coefficients, $\Delta\hat c = f(M) - f(M')$, with the true change $\Delta c^*$, using $\|\Delta\hat c - \Delta c^*\|_J$.

This term isolates the polarization that one part of a molecule induces in another. It is the signal a local model is structurally unable to fit and a global model should learn. Weight sharing matters for the familiar Siamese reason going back to Signature Verification Using a Siamese Time Delay Neural Network: the difference is taken between two outputs of one function, so shared errors cancel and only the response is supervised. Unlike the contrastive Siamese losses used for retrieval, this is a regression on differences. Because it has no negative pairs, the collapse to constant representations analyzed in Understanding Collapse in Non-contrastive Siamese Representation Learning cannot occur: the absolute loss $\mathcal{L}_J$ anchors the outputs.

Geometric symmetry ambiguity enters as a diagnostic. The vision literature shows that regression toward several equally valid targets produces their average, a failure called "mean collapse" (On Object Symmetries and 6D Pose Estimation from Images). For closed-shell molecules the density is a unique, gauge-free target, so we do not expect this failure broadly, and we do not claim it. It can appear in molecules with small HOMO–LUMO gaps (the energy gap between the highest occupied and lowest unoccupied orbitals). There, different starting guesses converge to different SCF solutions, and a network trained on whichever solution the reference run found may predict an average that sits between basins. We stratify all results by gap and report whether the remaining failures concentrate there. If they do, a multi-hypothesis density head in the spirit of the multi-head sector regressors of Pitteri et al. is the natural follow-up.

Baselines

The naive reference point is the standard superposition-of-atomic-densities (SAD) guess used by default in quantum chemistry codes. It defines a relative iteration count (RIC) of 100% and costs essentially nothing. Every learned method must beat it after its own inference time is included in wall-clock cost.

The strongest existing method is NequIP-L from Towards a Transferable Acceleration Method for Density Functional Theory. On SCFbench it reports a relative iteration count of approximately 66%, nearly constant across molecule sizes, which is an average 33.3% reduction. The same paper reports a ground-truth-density ceiling of about 60% reduction. We retrain NequIP-L with the authors' recipe on the same 43,862-molecule training set and the same DFT settings, and we confirm we reproduce the ~66% before comparing anything.

It is the right competitor because it already solves the size-transfer problem that sinks Hamiltonian targets. Our proposal only matters if it closes the remaining gap. Beating it means a statistically reliable RIC reduction on the 1,050-molecule out-of-distribution set, specifically reaching 58% or lower (at least a third of the way to the ceiling). That improvement must also be larger on the 26–60-atom molecules than on the in-distribution molecules.

We also report the Hamiltonian-target baseline (QHNet, from QH9) only to reproduce the known collapse on larger molecules, reported to reach 179% RIC. We do not treat it as a real competitor.

The third baseline is full softmax global attention of the AllScAIP type, placed on the same density backbone. This isolates whether global communication, rather than linearity, is what helps.

Ablations turn each ingredient on alone and in pairs: the Coulomb loss alone on NequIP-L, global attention alone with the original loss, and the Siamese response term alone. A data-matched control adds every fragment structure to the baseline's training set as an ordinary example, trained with the ordinary loss. This separates the benefit of the pair objective from the benefit of extra DFT labels.

All models get the same parameter budget within 10%, the same number of training steps, and the same hyperparameter search budget (the same number of trials over learning rate and loss weights). All are evaluated with the same SCF solver, convergence thresholds, and acceleration settings.

Why not just…?

Why not just make the local network deeper or give it a larger cutoff? This is the cheapest alternative, and at 60 atoms a big enough cutoff almost covers the molecule. AllScAIP's stretching experiments show local models failing as distance grows, but on potentials rather than SCF warm starts. So we test it directly with a NequIP-L variant whose cutoff and depth match our model's receptive field and parameter count. If that variant matches our out-of-distribution RIC, the attention stage is unnecessary at SCFbench scale. We would then report that result and restrict the architectural claim to the larger-molecule test below.

Why not just use full softmax attention, since 60 atoms is tiny? This is a real threat on SCFbench itself, where quadratic cost is negligible. We expect softmax and linear attention to perform similarly there, and if softmax is clearly better we will say so. Linear cost earns its place only in the regime AllScAIP identifies as memory-limited, and in the regime where SCF itself is most expensive. We therefore add a scaling test on larger molecules, where we measure memory, inference time and RIC for both. No reported result was found on Lacuna for SCF warm starts beyond 60 atoms.

Why not just change the loss and keep the architecture? The Coulomb-metric loss might recover most of the gain on its own by making the local model spend capacity on charge balance. The loss-only ablation settles this. Our hypothesis predicts a partial gain that shrinks with molecule size, because a local model cannot represent polarization from beyond its cutoff however it is trained. A flat or growing gain would falsify the long-range explanation.

Why not just train on iteration count directly with a differentiable SCF loop? That targets the true objective, but it requires backpropagating through many diagonalizations per example. At 44k molecules this is far more expensive, and gradients through near-degenerate eigenproblems are unstable. The Coulomb loss is our cheap stand-in. As a check, we measure on a held-out subset how well $\mathcal{L}_J$ predicts iteration count compared with per-coefficient error. If it does not correlate better, the stand-in is wrong.

Proposed Experiments

The primary experiment trains all models on SCFbench's 43,862 small molecules (up to 20 atoms; H, C, N, O, F, P, S) and evaluates RIC and convergence-failure rate on both the in-distribution test set and the 1,050 larger molecules. We plot RIC against atom count. The decisive signature is a slope: the baseline's RIC is reported flat at ~66%, and our model's gain over it should increase with size. We also report Coulomb-metric density error, dipole error, and total wall-clock time including inference.

Siamese pairs are generated by cutting terminal fragments farther than the local cutoff from the retained region. We label roughly one fragment per training molecule, a DFT cost well below that of the original set because fragments are smaller. These same labels are given to the data-matched control.

The mechanism check asks whether the gains come from the modes we claim. We project each model's initial density error onto the eigenvectors of the Coulomb matrix $J$, ordered by eigenvalue (the high-eigenvalue end corresponds to long-wavelength charge distributions), and show that our model's advantage lives at the long-wavelength end.

The scaling check runs softmax and linear variants on molecules of a few hundred atoms. These are drawn from larger organic structures that we compute ourselves with the same DFT protocol, since the file lists no existing benchmark at that size.

The symmetry diagnostic splits all test molecules by HOMO–LUMO gap and by point-group symmetry. For the smallest-gap group, it compares the warm-start density against densities converged from several perturbed SAD starts, to detect predictions lying between distinct SCF solutions.

The project succeeds if the full model reaches an out-of-distribution RIC clearly below 66%, beats the loss-only and deep-local ablations with a gap that grows with molecule size, and matches softmax attention on SCFbench while scaling to molecules where softmax does not fit in memory.