Strongly correlated molecules, such as a molecule whose bonds are being pulled apart, defeat cheap quantum-chemistry methods. Exact methods handle them, but at a cost that grows exponentially. A middle path builds the wavefunction from a handful of simple "mean-field" states: each describes the electrons as roughly independent but is allowed to break a physical symmetry, such as total spin. These states are then mixed so that the symmetry is restored. Because the states overlap rather than being orthogonal, the mixing step is a generalized eigenvalue problem, $Hc = ESc$, where $S$ is the matrix of overlaps between the states. This approach is called non-orthogonal configuration interaction (NOCI). Its accuracy depends almost entirely on which states are chosen, and today that choice is made by hand, by a uniform grid over a constraint parameter, or by greedy ranking from a fixed pool. Each variant comes with its own heuristics and none has convergence guarantees. The pool-based greedy variant is not covered by any unified convergence theory. All of them fight an overlap matrix that becomes nearly singular as states pile up.
We propose to treat NOCI basis growth as an optimization algorithm in the sense of the unification literature, specifically as a fully corrective conditional-gradient (Frank–Wolfe) method. The "atoms" are single Slater determinants (antisymmetrized products of orbitals, the simplest many-electron wavefunctions) on the continuous manifold of such determinants. The "linear minimization step" picks the determinant that most reduces a provable lower bound on the energy decrease. The "correction step" is the exact generalized eigenvalue solve. The key ingredient is a selection score that divides the new state's coupling to the current residual by its distance from the span of states already chosen. One rule then gives three things: a per-step energy-decrease guarantee, an explicit lower bound on how well-conditioned $S$ stays, and a single algorithm of which spin-GCM, the greedy ADAPT-GCIM pool search, and orthogonal coordinate-descent CI are restricted special cases.
Core Claim
Let $\psi_k$ be the current NOCI solution with energy $E_k$ in the span $V_k$ of $k$ determinants. Let $r_k = (H - E_k)\psi_k$ be its residual, and let $P_k$ be the projector onto $V_k$ in the metric defined by $S$. Adding a determinant $\phi$ and re-solving the resulting two-dimensional Rayleigh–Ritz problem lowers the energy by at least
$$ E_k - E_{k+1} \;\ge\; \frac{|\langle \phi | r_k\rangle|^2}{\Lambda\,\langle \phi|(1-P_k)|\phi\rangle}, $$
where $\Lambda$ is an upper bound on the spectral spread of $H$ above $E_k$. This is a standard consequence of the two-by-two secular equation. The right-hand side is a surrogate in the sense of Optimization with First-Order Surrogate Functions, which unifies proximal gradient, block coordinate descent and Frank–Wolfe as minimizations of surrogates that bound the true objective. We use its majorization–minimization template, applied here to a generalized eigenproblem rather than a finite-sum loss.
We propose to choose each new determinant by maximizing the surrogate score $|\langle\phi|r_k\rangle|^2/\langle\phi|(1-P_k)|\phi\rangle$ over the full manifold of determinants. We parameterize that manifold by Thouless rotations of the orbitals, which allow unrestricted spin and, optionally, generalized spin orientation. Every quantity in the score reduces to overlaps and transition matrix elements between single determinants. The generalized Wick theorem makes these polynomial-cost, as noted in Say NO to Optimization. The method therefore stays entirely classical and avoids the exponential cost that correlated non-orthogonal states incur.
The denominator does two jobs. It rewards directions genuinely new relative to $V_k$. Because the diagonal entries of the Cholesky factor of $S$ are exactly these residual distances, rejecting candidates whose distance falls below $\delta$ bounds the smallest eigenvalue of $S$ from below as a function of $\delta$ and $k$. This replaces the ad hoc singular-value truncation used in A Non-Orthogonal Variational Quantum Eigensolver with a criterion built into selection.
On theory, we expect an $O(1/k)$ energy-error rate whenever the target state has bounded "atomic norm" over the determinant manifold. That is the standard greedy-approximation and Frank–Wolfe rate, which follows from the per-step bound above. The rate degrades gracefully if the inner maximization is only approximate. We will state and prove this carefully, including the approximate-selection case, rather than assume it.
How This Relates to Existing Work
Three non-orthogonal subspace methods motivate the gap.
Spin-generator coordinate method for electronic structure and the 2-site Hubbard study generate states by scanning a spin-constraint multiplier on a grid and mixing them. They report convergence around $n=7$ states, but the grid is fixed in advance along one generator coordinate. Our selection searches the whole determinant manifold and contains their one-parameter family as a subset.
Unleashed from constrained optimization (GCIM/ADAPT-GCIM) grows a non-orthogonal basis by ranking gradients from a finite pool of coupled-cluster excitation operators. The research file notes it lacks a convergence theory and suffers degrading conditioning of $S$. Our score differs in two ways: it is normalized by distance from the current span, and its candidates are continuous mean-field states rather than a discrete operator pool.
Say NO to Optimization builds its basis statically from unrestricted Hartree–Fock references plus MP2-parameterized coupled-cluster dressing. It reports residual spin contamination in the $Q_1$ state of square H$_4$, which we treat as a specific failure to test against.
Coordinate Descent Full Configuration Interaction is our closest algorithmic relative. It casts CI as coordinate optimization over orthogonal determinants with frequency-weighted updates and hard thresholding. In our formulation it is the special case where the dictionary is restricted to orthogonal occupation-number determinants and the denominator is identically one. We therefore expect it to need many more atoms for static correlation, which is the regime where breaking symmetry pays off.
One constraint from outside chemistry shapes the inner optimizer. Improving Optimization for Models With Continuous Symmetry Breaking shows that gradient descent stalls along flat directions created by continuous symmetries. When $\psi_k$ is a spin eigenstate, the selection score is exactly invariant under spin rotations of $\phi$, so the inner maximization has an exact flat orbit. We will fix that gauge explicitly, following their argument, rather than let the optimizer drift.
We do not claim to fix the dynamic-correlation deficit that spin-GCM documents. Where that matters we add second-order perturbation theory (NOCI-PT2) on top, exactly as they do.
Research Program
The first deliverable is the theory. We will derive the surrogate bound, the conditioning guarantee on $S$, and the convergence rate under exact and approximate selection. We will then show formally how spin-GCM, ADAPT-GCIM restricted to determinant-valued generators, and CDFCI arise as dictionary restrictions or score simplifications.
The second deliverable is a classical implementation. The outer loop is the generalized eigenvalue solve. The inner loop is a gauge-fixed quasi-Newton maximization of the score over Thouless parameters, using analytic gradients from generalized Wick contractions. Each score evaluation costs $O(k)$ determinant-pair matrix elements, so building a basis of $M$ states costs $O(M^2)$ pair evaluations, the same order as assembling the NOCI matrices themselves.
The third deliverable is an excited-state extension. It applies the same selection to the residual of the second Ritz vector while holding the shared span, which directly targets the H$_4$ spin-contamination failure.
Baselines
The naive reference is the single best symmetry-broken mean-field state (unrestricted Hartree–Fock), plus NOCI built from whatever unrestricted solutions a standard solver finds spontaneously. Any proposed method must clearly beat this floor near bond-breaking.
The strongest same-family competitor is spin-GCM with a uniform constraint grid, from Spin-generator coordinate method for electronic structure. It reports recovering the full CI energy of H$_2$ in STO-3G "with just a few c-UHF states," and convergence of NOCI($n$) around $n=7$. In cc-pVDZ, it reports that more than 90% of the correlation energy needs NOCI-PT2. We run it with the same integrals and the same budget of determinants. Beating it means reaching lower energy at equal $M$, or matching it with fewer states, on H$_2$ in cc-pVDZ, HeH$^+$ in 6-31G and square H$_4$. On the 2-site Hubbard model, where the Hubbard study reports that NOCI(4) is exact, we must also reach exactness by four states.
The second strong competitor is ADAPT-GCIM from Unleashed from constrained optimization. It reports state overlaps above $1-10^{-15}$ for nearly all tested molecules, and 12–13 seconds on a laptop for stretched H$_6$ against about 4 hours for VQE. We run it classically with a statevector representation on H$_4$, H$_6$, LiH and BeH$_2$, at matched wall-clock time. We compare the energy error as a function of basis size and the condition number of $S$ along the trajectory. Because its correlated basis states are richer than single determinants, matching its accuracy is not our goal. The claim to test is that we reach a given error with a better-conditioned $S$ and polynomial classical cost, on systems where statevector ADAPT-GCIM is infeasible.
For NOQE, Say NO to Optimization reports chemical accuracy (1 kcal/mol) for the S$_0$ and T$_1$ states of H$_2$ at larger distances, and T$_1$ of square H$_4$ within chemical accuracy. It also reports $Q_1$ spin contamination at short distances. Beating it means a spin-pure $Q_1$, with $\langle S^2\rangle$ within $10^{-3}$ of 6, at matched basis size.
CDFCI from Coordinate Descent Full Configuration Interaction serves both as the reference energy source and as the orthogonal-dictionary baseline. It reports benchmarking the N$_2$ cc-pVDZ binding curve to $10^{-3}$ mHa accuracy. We compare the number of atoms each method needs to recover a fixed fraction of the static correlation energy along that curve.
The key ablation keeps everything identical but drops the denominator. It selects by raw residual overlap and handles near-dependence afterward by singular-value truncation of $S$ at a tuned threshold. This isolates whether the normalized surrogate, rather than the continuous search, carries the gains. A second ablation restricts the search to the one-dimensional spin-constraint family. That separates the value of a richer dictionary from the value of the selection rule.
Why not just…?
Why not just use selected or coordinate-descent CI in an orthogonal basis? Orthogonal methods never face an ill-conditioned $S$ and come with mature tooling. For dynamic correlation in compact molecules they may simply win. The evidence in the file points the other way only for static correlation. Spin-GCM and the Hubbard study recover near-exact energies with two to seven non-orthogonal states, whereas CDFCI's strongly correlated benchmarks (N$_2$, Cr$_2$) traverse very large determinant spaces. Our atom-count comparison along the N$_2$ dissociation curve measures this directly. If the orthogonal method needs comparable numbers of atoms at stretched geometries, the proposal's premise fails, and we will report that.
Why not just run ADAPT-style greedy selection and truncate small singular values of $S$? This is the real threat, since it is what practitioners already do. The first ablation is designed to settle it. If raw-overlap selection plus truncation matches the normalized rule at equal $M$ and equal compute, then the conditioning guarantee is cosmetic. The contribution would shrink to the theory plus the continuous dictionary. We expect the normalized rule to help most where truncation discards states carrying needed spin components, which is the H$_4$ $Q_1$ regime. That is a falsifiable prediction.
Why not just optimize all $M$ determinants jointly in one nonlinear variational problem? Joint optimization can, in principle, find a better $M$-state basis than greedy growth, because greedy choices are never revisited. But it is a high-dimensional nonconvex problem with many symmetry-related minima. It is exactly the kind of landscape whose trainability problems motivated GCIM and NOQE in the first place. We will run joint optimization from the greedy solution and from random starts, under the same total compute. If joint refinement substantially improves on greedy at fixed $M$, we will adopt it as a final polishing stage rather than claim greedy is sufficient.
Why not just project a single broken-symmetry determinant onto good spin? Large-scale self-consistent nuclear mass calculations reports that approximate restoration breaks down near closed-shell nuclei, with errors exceeding 1 MeV. Spin-Symmetry Projected constrained UHF is the chemistry analogue of a single projected determinant. We include projected unrestricted Hartree–Fock as an additional reference curve, to show where mixing several states is needed beyond projection alone.
Proposed Experiments
All energies are compared against full CI, computed exactly for small systems and taken from CDFCI-quality references for N$_2$. The primary plot is energy error versus number of determinants $M$. Secondary metrics are the condition number of $S$ along the trajectory, $\langle S^2\rangle$ of each final state, and wall-clock time.
The systems are the 2-site Hubbard model across $U/t$; H$_2$ in STO-3G and cc-pVDZ; HeH$^+$ in 6-31G; square H$_4$ in STO-3G and 6-31G for the S$_0$, T$_1$ and Q$_1$ states; linear H$_6$ at stretched geometry; and N$_2$ in cc-pVDZ along the dissociation curve.
Three outcomes are decisive. First, the normalized rule must beat the truncation ablation at equal $M$ on H$_4$ and N$_2$. Second, the excited-state extension must give a spin-pure Q$_1$ state where NOQE is contaminated. Third, the observed error-versus-$M$ curves must be consistent with the derived greedy rate. If the curves are flat where the theory predicts steady decrease, or the ablation ties, the unifying view has explanatory value but no practical edge, and the note's central claim is falsified.
References
- Optimization with First-Order Surrogate Functions
- Say NO to Optimization: A Nonorthogonal Quantum Eigensolver
- A Non-Orthogonal Variational Quantum Eigensolver
- Spin-generator coordinate method for electronic structure
- Spin-constrained Hartree-Fock and the generator coordinate method for the 2-site Hubbard model
- Unleashed from constrained optimization: quantum computing for quantum chemistry employing generator coordinate inspired method
- Coordinate Descent Full Configuration Interaction
- Improving Optimization for Models With Continuous Symmetry Breaking
- Large-scale self-consistent nuclear mass calculations
- Spin-Symmetry Projected Constrained Unrestricted Hartree–Fock