Geometry of the sample frequency spectrum and the perils of demographic\n inference

Zvi Rosen, Anand Bhaskar, Sébastien Roch, Yun S. SongarXiv 2017 · DOI · arXiv

Abstract

The sample frequency spectrum (SFS), which describes the distribution of mutant alleles in a sample of DNA sequences, is a widely used summary statistic in population genetics. The expected SFS has a strong dependence on the historical population demography and this property is exploited by popular statistical methods to infer complex demographic histories from DNA sequence data. Most, if not all, of these inference methods exhibit pathological behavior, however. Specifically, they often display runaway behavior in optimization, where the inferred population sizes and epoch durations can degenerate to 0 or diverge to infinity, and show undesirable sensitivity of the inferred demography to perturbations in the data. The goal of this paper is to provide theoretical insights into why such problems arise. To this end, we characterize the geometry of the expected SFS for piecewise-constant demographic histories and use our results to show that the aforementioned pathological behavior of popular inference methods is intrinsic to the geometry of the expected SFS. We provide explicit descriptions and visualizations for a toy model with sample size 4, and generalize our intuition to arbitrary sample sizes n using tools from convex and algebraic geometry. We also develop a universal characterization result which shows that the expected SFS of a sample of size n under an arbitrary population history can be recapitulated by a piecewise-constant demography with only k(n) epochs, where k(n) is between n/2 and 2n-1. The set of expected SFS for piecewise-constant demographies with fewer than k(n) epochs is open and non-convex, which causes the above phenomena for inference from data.

Introduction

Demographic inference is a cornerstone of modern population genetics, allowing researchers to reconstruct the history of populations from DNA sequence data. The primary tool for this is the Sample Frequency Spectrum (SFS), which summarizes the distribution of mutant alleles. However, popular inference methods often exhibit "pathological" behavior, such as returning infinite population sizes or epochs of zero duration. This paper provides a rigorous mathematical explanation for these failures, demonstrating that they are intrinsic to the geometry of the expected SFS.

Methods

The authors move beyond simple statistical modeling into the realm of geometry to understand these instabilities:

  • Coalescent Theory: They use the standard mathematical framework for how gene lineages merge backward in time.
  • Algebraic Geometry: Concepts like secant varieties and Zariski closures are used to determine the dimensionality and boundaries of SFS manifolds.
  • Convex Geometry: The authors apply Carathéodory’s Theorem to prove how many "epochs" (periods of constant population size) are needed to represent any arbitrary population history.

Geometric Characterization of the SFS

A key finding is the Universal Recapitulation Theorem, which states that any arbitrary population history can be perfectly represented by a piecewise-constant demography with $k(n)$ epochs, where $n/2 \le k(n) \le 2n-1$. However, for a fixed number of epochs, the set of expected SFS vectors is both non-convex and "open."

To build intuition, the authors provide explicit descriptions for a sample size of $n=4$. The normalized SFS manifold in this case can be visualized in three dimensions, revealing the complex structure of the space of possible genetic patterns.

Geometric visualization of the SFS manifold for sample size n=4, showing the regions that define the boundaries of expected genetic patterns.

Why Inference Fails: Non-Convexity and Openness

The "openness" of the SFS manifold is the primary driver of pathological behavior. An open set does not contain its own boundaries. When empirical data—which inevitably contains noise—falls outside this open set, optimization algorithms attempting to find the "best fit" will naturally diverge toward the boundary. These boundaries correspond to degenerate demographic parameters, such as infinite population sizes or zero-length epochs.

Furthermore, the non-convexity of the manifold leads to extreme sensitivity. Small perturbations in the data can cause the "closest point" on the manifold to jump between radically different regions, resulting in wildly inconsistent demographic reconstructions.

Simulated trajectories in the SFS simplex illustrating how small perturbations in data can lead to unstable demographic inferences and runaway behavior toward the manifold boundaries.

Conclusion and Future Directions

This geometric perspective reveals that the instabilities in demographic inference are not merely software bugs but are fundamental properties of the SFS itself. While the study focuses on piecewise-constant demographies, the authors show these can approximate any history. Future work may leverage these geometric bounds to develop new, non-parametric "tests of neutrality" that do not rely on specific demographic assumptions, potentially identifying data that falls outside the "universal" SFS manifold for any possible neutral history.

Summary updated

Abstract

The sample frequency spectrum (SFS), which describes the distribution of\nmutant alleles in a sample of DNA sequences, is a widely used summary statistic\nin population genetics. The expected SFS has a strong dependence on the\nhistorical population demography and this property is exploited by popular\nstatistical methods to infer complex demographic histories from DNA sequence\ndata. Most, if not all, of these inference methods exhibit pathological\nbehavior, however. Specifically, they often display runaway behavior in\noptimization, where the inferred population sizes and epoch durations can\ndegenerate to 0 or diverge to infinity, and show undesirable sensitivity of the\ninferred demography to perturbations in the data. The goal of this paper is to\nprovide theoretical insights into why such problems arise. To this end, we\ncharacterize the geometry of the expected SFS for piecewise-constant\ndemographic histories and use our results to show that the aforementioned\npathological behavior of popular inference methods is intrinsic to the geometry\nof the expected SFS. We provide explicit descriptions and visualizations for a\ntoy model with sample size 4, and generalize our intuition to arbitrary sample\nsizes n using tools from convex and algebraic geometry. We also develop a\nuniversal characterization result which shows that the expected SFS of a sample\nof size n under an arbitrary population history can be recapitulated by a\npiecewise-constant demography with only k(n) epochs, where k(n) is between n/2\nand 2n-1. The set of expected SFS for piecewise-constant demographies with\nfewer than k(n) epochs is open and non-convex, which causes the above phenomena\nfor inference from data.\n
BibTeX
@misc{rosen2017geometry,
  author = {Zvi Rosen and Anand Bhaskar and Sébastien Roch and Yun S. Song},
  title = {{Geometry of the sample frequency spectrum and the perils of demographic\textbackslash\{\}n inference}},
  year = {2017},
  eprint = {1712.05035},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1712.05035},
}
Figure