Heritability of complex traits in sub-populations experiencing bottlenecks and growth

Cameron S. Taylor, Daniel J. Lawson

Abstract: Populations that have experienced a bottleneck are regularly used in Genome Wide Association Studies (GWAS) to investigate variants associated with complex traits. It is generally understood that these isolated sub-populations may experience high frequency of otherwise rare variants with large effect size, and therefore provide a unique opportunity to study said trait. However, the demographic history of the population under investigation affects all SNPs that determine the complex trait genome-wide, changing its heritability and genetic architecture. We use a simulation based approach to identify the impact of the demographic processes of drift, expansion, and migration on the heritability of complex trait. We show that demography has considerable impact on complex traits. We then investigate the power to resolve heritability of complex traits in GWAS studies subjected to demographic effects. We find that demography is an important component for interpreting inference of complex traits and has a nuanced impact on the power of GWAS. We conclude that demographic histories need to be explicitly modelled to properly quantify the history of selection on a complex trait.

Introduction: The Dynamic Nature of Heritability

Genome-Wide Association Studies (GWAS) frequently target "bottlenecked" populations—such as the Ashkenazi Jews, Finns, or Icelanders—under the premise that their reduced genetic diversity simplifies the identification of rare variants with large effects. However, this approach often overlooks a fundamental biological reality: heritability ($h^2$) is not a fixed physical constant. Instead, it is a dynamic ratio of genetic variance to total phenotypic variance, heavily influenced by a population's demographic history.

When a population undergoes a severe contraction (a bottleneck), it loses a significant portion of its genetic "fuel." This study addresses a critical gap in genomic research: how the history of population contraction, subsequent growth, and migration alters the overall heritability and genetic architecture of complex traits. Ignoring these histories can lead to fundamentally flawed inferences about how natural selection has shaped human traits.

Methodology: Simulating Synthetic Genealogies

To investigate these effects, researchers employed a large-scale simulation-based approach using msprime, a high-performance coalescent simulator. This allowed for the generation of thousands of synthetic human histories with controlled variables.

  1. Demographic Modeling: The study compared a large, stable "Reference" population against a "Bottlenecked" population. The latter was simulated to split from the reference, undergo a sharp reduction in size ($N$), and then experience varying degrees of subsequent growth or migration.
  2. Trait Simulation: Causal effects were assigned to specific genetic markers (SNPs). The model coupled a variant’s effect size to its frequency, ensuring that rare variants were more likely to have large effects, thereby mimicking the signature of negative selection.
  3. Quantifying Drift: By comparing genetic variance between the groups, the authors quantified how much "detectable" heritability shifted due to random genetic drift and the subsequent power of GWAS to resolve these traits.

Shifting Genetic Architectures

A central finding of the research is that population bottlenecks do not merely reduce diversity; they fundamentally reshape the relationship between a variant's frequency and its impact on a trait.

Figure 1: (a) Comparison of SNP effect sizes and allele frequencies between populations. (b) Coalescent tree representing demographic history.

As illustrated in Figure 1a, variants that are rare and possess large effect sizes in a stable Reference Population (red) can drift to significantly higher frequencies in a Bottlenecked Population (blue). This "magnification" effect makes specific variants easier to detect in GWAS. However, as shown in the coalescent tree (Figure 1b), the shared ancestry within the bottlenecked group means that while some variants are amplified, many others are lost entirely, leading to a more volatile genetic landscape.

Key Insights and Results

The Variance Spike

While a bottleneck generally decreases the average heritability of traits due to the loss of variants, it drastically increases the variance of heritability across different traits. In a drifted population, one trait might randomly become much more heritable because a large-effect variant drifted to high frequency, while another trait’s heritability might vanish. This makes isolated sub-populations "noisy" proxies for broader human populations.

Migration as a Buffer

The study found that even minimal migration—as little as "one individual per generation" from a larger reference population—is sufficient to significantly offset the effects of genetic drift on heritability. This suggests that even slightly open populations maintain a much more stable genetic architecture than completely isolated ones.

Time and Recovery

The variance in heritability increases almost linearly with the time elapsed since the bottleneck. While rapid population growth following a "squeeze" helps recover some heritability, the final genetic state remains dominated by the severity of the initial bottleneck. In simulations where the reference population had $h^2 = 0.5$, severe bottlenecks (reducing the population to 1,000 individuals) typically dropped the mean heritability to approximately 0.4.

Conclusion and Future Directions

The research demonstrates that for traits with lower "true" heritability, GWAS detection power in bottlenecked populations can drop significantly because the bottleneck often eliminates the "intermediate" variants that provide the bulk of the statistical signal.

Future work must address the "inverse problem": while it is straightforward to simulate forward from history to heritability, it remains difficult to reconstruct ancestral genetic architectures from modern data. The authors conclude that to truly understand the history of selection on human traits, researchers must explicitly incorporate demographic modeling into GWAS pipelines rather than treating all populations as genetically equivalent.

Summary updated

Abstract

Populations that have experienced a bottleneck are regularly used in Genome Wide Association Studies (GWAS) to investigate variants associated with complex traits. It is generally understood that these isolated sub-populations may experience high frequency of otherwise rare variants with large effect size, and therefore provide a unique opportunity to study said trait. However, the demographic history of the population under investigation affects all SNPs that determine the complex trait genome-wide, changing its heritability and genetic architecture. We use a simulation based approach to identify the impact of the demographic processes of drift, expansion, and migration on the heritability of complex trait. We show that demography has considerable impact on complex traits. We then investigate the power to resolve heritability of complex traits in GWAS studies subjected to demographic effects. We find that demography is an important component for interpreting inference of complex traits and has a nuanced impact on the power of GWAS. We conclude that demographic histories need to be explicitly modelled to properly quantify the history of selection on a complex trait.
BibTeX
@article{taylor2024heritability,
  author = {Cameron S. Taylor and Daniel J. Lawson},
  title = {{Heritability of complex traits in sub-populations experiencing bottlenecks and growth}},
  journal = {Journal of Human Genetics},
  year = {2024},
  doi = {10.1038/s10038-024-01249-2},
}
Figure