Abstract
Computing averages over a target probability density by statistical re-weighting of a set of samples with a different distribution is a strategy which is commonly adopted in fields as diverse as atomistic simulation and finance. Here we present a very general analysis of the accuracy and efficiency of this approach, highlighting some of its weaknesses. We then give an example of how our results can be used, specifically to assess the feasibility of high-order path integral methods. We demonstrate that the most promising of these techniques—which is based on re-weighted sampling—is bound to fail as the size of the system is increased, because of the exponential growth of the statistical uncertainty in the re-weighted average.
Introduction
In computational physics and chemistry, calculating averages over a target probability distribution $p$ is often achieved by sampling a simpler distribution $p_0$ and applying a correction weight $w = p/p_0$. This re-weighted sampling is particularly attractive in Path Integral Molecular Dynamics (PIMD) to implement high-order integration schemes, such as the Suzuki-Chin (SC) propagator, without the prohibitive cost of calculating the Hessian matrix. However, this study investigates whether this computational shortcut remains viable as system size increases.
The Statistical Threshold of Re-weighting
The efficiency of re-weighted estimators is fundamentally linked to the fluctuations of the log-weights $h = -\ln w$. Through asymptotic analysis, we demonstrate that the statistical error and systematic bias of the re-weighted average depend critically on the variance $\sigma^2(h)$.
As shown in Figure 1, when $\sigma(h) \lesssim 1$, the error follows the standard $1/\sqrt{n}$ scaling. However, once $\sigma(h)$ exceeds this threshold, the statistical uncertainty explodes, and the estimator enters a non-asymptotic regime where the perceived convergence can be highly misleading. This "insidious" error makes it difficult for researchers to detect when their simulations have failed to converge.
The Curse of System Size
Because energy is an extensive property, the variance of the difference Hamiltonian—and thus the variance of the log-weights—scales linearly with the number of particles $N$. Consequently, the statistical efficiency of re-weighting decays exponentially with system size. This "curse of system size" implies that high-order methods relying on re-weighting will inevitably fail for large-scale condensed-phase simulations.
Benchmark: Liquid Water Simulation
To validate these theoretical findings, we compared standard Trotter integration, re-weighted Suzuki-Chin (rw-SC), and Path Integrals with Generalized Langevin Equations (PI+GLE) for a system of 216 flexible water molecules at 298 K.
Figure 2 illustrates that while rw-SC aims for high-order accuracy, its statistical error in a large system like 216 water molecules is significantly higher than that of the simpler Trotter scheme. Our analysis suggests that for liquid water under ambient conditions, re-weighted high-order methods are only advantageous for systems smaller than approximately 100 molecules. In contrast, the PI+GLE method achieves rapid convergence without the sampling penalty associated with re-weighting.
Conclusion
While re-weighted sampling remains a useful tool for small clusters or mildly quantum systems, it is fundamentally unsuitable for large-scale atomistic simulations. For condensed-phase problems, researchers should favor stochastic acceleration methods like PI+GLE, which provide high-order convergence benefits without the exponential growth of statistical uncertainty.
Summary updated