On estimating the structure factor of a point process, with applications to hyperuniformity

Diala Hawat, Guillaume Gautier, Rémi Bardenet, Raphaël Lachièze-ReyStat. Comput 2023 · DOI

Introduction

Hyperuniformity is a concept bridging statistical physics and machine learning, describing a state of matter or a mathematical point process where large-scale density fluctuations are suppressed. In a hyperuniform system, the variance of the number of points within a large observation window grows slower than the volume of that window. This property is highly sought after; in physics, it leads to unique optical properties, while in statistics, it enables "super-efficient" Monte Carlo integration.

The primary diagnostic for hyperuniformity is the structure factor $S(k)$. A process is defined as hyperuniform if $S(k)$ approaches zero as the wavenumber $k$ tends toward zero. Despite its importance, estimating $S(k)$ accurately has historically been difficult, often relying on field-specific heuristics rather than standardized statistical theory. This paper provides a rigorous framework to address these challenges, introducing a comprehensive survey of estimators, a new Python toolbox, and the first asymptotically valid statistical test for hyperuniformity.

The Estimation Challenge

Estimating the structure factor from finite data is notoriously sensitive to bias. Researchers across physics, chemistry, and spatial statistics have traditionally used different estimators based on "field lore." Furthermore, the field has lacked formal statistical tests, with researchers often relying on visual inspection of plots or arbitrary indices like the $H$-index to determine if a process is hyperuniform.

The authors identify two primary families of estimators:

  1. Stationary Estimators: These include the "scattering intensity" common in physics and its tapered or multitapered variants borrowed from time-series analysis.
  2. Isotropic Estimators: These assume the process is rotation-invariant, utilizing Bartlett’s isotropic estimator and numerical quadratures of Hankel transforms (such as Ogata’s and Baddour-Chouinard’s methods).

Methodological Contributions

1. The structure_factor Toolbox

To standardize empirical investigations, the authors released an open-source Python library, structure_factor. This library implements a wide array of estimators and tests, providing a unified platform for researchers to analyze point processes.

2. A New Statistical Test

Moving beyond visual heuristics, the paper introduces a multiscale statistical test. By leveraging a debiasing technique (coupled sum estimator) by Rhee and Glynn (2015), the authors construct a confidence interval for $S(0)$. If zero is contained within this interval, the null hypothesis of hyperuniformity is not rejected. This provides a rigorous mathematical basis for identifying hyperuniform states.

3. Tapering and Allowed Wavevectors

A significant source of error in structure factor estimation is the bias caused by the finite observation window. The authors demonstrate that evaluating the scattering intensity only at specific "allowed wavevectors"—where the window's Fourier transform is zero—is essential for an unbiased estimate. Furthermore, they show that "tapering" (weighting points based on their distance from the window edge) significantly reduces variance. Multitapering, which averages over several orthogonal tapers, emerged as the most effective approach for general stationary processes.

Experimental Validation

The authors benchmarked these tools on several classic point processes:

  • The Ginibre Ensemble: A strongly hyperuniform, stationary, and isotropic process (Class I).
  • The KLY Process: A hyperuniform process that is stationary but not isotropic.
  • The Homogeneous Poisson Process: A baseline for complete randomness (non-hyperuniform).
  • The Thomas Point Process: A clustered, non-hyperuniform process.

Key Results

  • Quantitative Performance: In Mean Square Error (MSE) comparisons, the directly debiased multitapered estimator was the superior choice for general stationary processes.
  • The Power of Isotropy: If a process is known to be isotropic (like the Ginibre ensemble), Bartlett’s isotropic estimator is vastly superior, providing integrated MSE values an order of magnitude smaller than Hankel transform-based methods.
  • Test Accuracy: The proposed multiscale test successfully identified hyperuniformity in the Ginibre and KLY processes while correctly rejecting it for the Poisson and Thomas processes at a 99.7% confidence level.

Limitations and Future Directions

While the paper establishes a new standard for the field, some challenges remain. Bartlett’s isotropic estimator, while accurate, has a computational complexity of $O(N^2)$, making it slower than the $O(N \log N)$ scattering intensity for very large datasets. Additionally, the statistical test faces difficulties when a process is "near-hyperuniform" (e.g., a Ginibre ensemble where a small percentage of points have been randomly removed).

Future research may focus on optimizing tapers specifically for point processes and developing more sensitive tests for near-hyperuniform states. Nevertheless, this work provides the necessary theoretical and practical tools to move hyperuniformity research from qualitative observation to quantitative rigor.

Summary updated

BibTeX
@article{hawat2023estimating,
  author = {Diala Hawat and Guillaume Gautier and Rémi Bardenet and Raphaël Lachièze-Rey},
  title = {{On estimating the structure factor of a point process, with applications to hyperuniformity}},
  journal = {Stat. Comput.},
  year = {2023},
  doi = {10.1007/s11222-023-10219-1},
}
Figure