Heritability of complex traits in sub-populations experiencing bottlenecks and growth

Cameron S. Taylor, Daniel J. Lawson

Abstract

Populations that have experienced a bottleneck are regularly used in Genome Wide Association Studies (GWAS) to investigate variants associated with complex traits. It is generally understood that these isolated sub-populations may experience high frequency of otherwise rare variants with large effect size, and therefore provide a unique opportunity to study said trait. However, the demographic history of the population under investigation affects all SNPs that determine the complex trait genome-wide, changing its heritability and genetic architecture. We use a simulation-based approach to identify the impact of the demographic processes of drift, expansion, and migration on the heritability of complex trait. We show that demography has considerable impact on complex traits. We then investigate the power to resolve heritability of complex traits in GWAS studies subjected to demographic effects. We find that demography is an important component for interpreting inference of complex traits and has a nuanced impact on the power of GWAS. We conclude that demographic histories need to be explicitly modelled in order to properly quantify the history of selection on a complex trait.

Introduction

Population history is not merely a background variable in genetic studies; it is a fundamental architect of trait heritability. Geneticists frequently target "bottlenecked" populations—such as Icelanders, Finns, or Ashkenazi Jews—because these groups descended from a small number of founders. In these isolated populations, rare genetic variants with large effects can "drift" to higher frequencies, making them significantly easier to detect via Genome-Wide Association Studies (GWAS).

However, the same demographic forces that amplify these specific variants also reshape the entire genomic landscape. This alters a trait's heritability ($h^2$), the proportion of phenotypic variation attributable to genetic factors. Current research often lacks a clear understanding of how specific demographic processes—drift, expansion, and migration—interact to shift heritability over time. Without accounting for these histories, researchers risk misinterpreting whether a trait's prevalence is due to natural selection or simply the result of a turbulent population history.

Methods

This study utilizes a simulation-based framework to explore these dynamics. Using msprime, a high-performance coalescent simulator, the researchers modeled the following:

  1. Genealogy Generation: A large "Reference" population was simulated alongside a smaller "Bottlenecked" sub-population that diverged at a specific time ($T$).
  2. Phenotype Modeling: Effect sizes were assigned to Single Nucleotide Polymorphisms (SNPs) under a model of "negative selection," where variants with large effects are typically maintained at low frequencies.
  3. Variable Testing: Parameters such as bottleneck severity, time since divergence, migration rates, and post-bottleneck growth rates were systematically varied.
  4. GWAS Simulation: To mirror real-world research, the study applied statistical "thresholding" to identify SNPs detectable within the power constraints of a typical GWAS.
Conceptual model of genetic architecture shifts following a population bottleneck.

The simulations generated synthetic genetic data representing 0.1 to 20 Morgans of the genome, designed to reflect human demographic histories such as the "Out-of-Africa" bottleneck (~2,000 generations ago) or the more recent Ashkenazi Jewish population expansion (~200 generations ago).

Key Insights and Results

The simulations revealed that demography fundamentally alters the "genetic architecture" of complex traits:

  • The "Drift Boost": Bottlenecks generally increase the heritability of complex traits. As rare variants with large effects are pushed to higher frequencies by random drift, the total genetic variance within the population increases. In simulations where the reference population had a heritability of 0.5, a severe bottleneck (reducing population to 10% of original size) caused heritability to jump to approximately 0.8.
  • Time and Volatility: While a bottleneck immediately raises average heritability, the variance of that heritability increases linearly with time. The longer a population remains isolated, the more unpredictable its genetic architecture becomes.
  • Growth and Migration: Rapid population expansion can paradoxically reduce heritability if large-effect alleles become "fixed" (present in everyone), thereby removing genetic variation. Conversely, even minimal migration from a larger reference population acts as a "reset button," quickly reverting the sub-population’s heritability to ancestral levels.
Impact of bottleneck size and timing on the distribution and variance of trait heritability.

Discussion

The study demonstrates that GWAS power is highly nuanced in the context of demography. While drifted populations can make high-effect variants easier to resolve, traits that happen to have low heritability in a specific sub-population may result in failed GWAS detection, even if the trait is highly genetic in the broader global population.

A significant limitation remains: while we can simulate forward from history to genetic architecture, we currently lack robust tools for inverse inference—reconstructing a population's complex demographic history solely from modern genetic data. The authors suggest that integrating Ancient DNA could provide the necessary temporal data to distinguish between the signatures of natural selection and the echoes of ancient population crashes. Ultimately, demographic histories must be explicitly modeled to accurately interpret the genetic basis of complex traits.

Summary updated

Abstract

1 Abstract Populations that have experienced a bottleneck are regularly used in Genome Wide Association Studies (GWAS) to investigate variants associated with complex traits. It is generally understood that these isolated sub-populations may experience high frequency of otherwise rare variants with large effect size, and therefore provide a unique opportunity to study said trait. However, the demographic history of the population under investigation affects all SNPs that determine the complex trait genome-wide, changing its heritability and genetic architecture. We use a simulation-based approach to identify the impact of the demographic processes of drift, expansion, and migration on the heritability of complex trait. We show that demography has considerable impact on complex traits. We then investigate the power to resolve heritability of complex traits in GWAS studies subjected to demographic effects. We find that demography is an important component for interpreting inference of complex traits and has a nuanced impact on the power of GWAS. We conclude that demographic histories need to be explicitly modelled in order to properly quantify the history of selection on a complex trait.
BibTeX
@article{taylor2024heritability,
  author = {Cameron S. Taylor and Daniel J. Lawson},
  title = {{Heritability of complex traits in sub-populations experiencing bottlenecks and growth}},
  journal = {Journal of Human Genetics},
  year = {2024},
  doi = {10.1038/s10038-024-01249-2},
}
Figure