Posterior predictive checks to quantify lack-of-fit in admixture models of latent population structure

David Mimno, David M. Blei, Barbara E. EngelhardtProceedings of the National Academy of Sciences 2015 · DOI

TL;DR

This paper introduces a novel framework using Bayesian posterior predictive checks (PPCs) to rigorously evaluate the fit of admixture models, commonly used to infer population structure in genetic data. By developing five specific "discrepancy functions," the authors demonstrate that the same admixture model can perform very differently across diverse genomic datasets, highlighting the critical need for model validation in population genetics.

What Problem Does This Paper Address?

Admixture models are a cornerstone of population genetics, used to understand the complex ancestry of individuals and to control for confounding factors in genetic association studies. These models represent genomes as mixtures of ancestral populations, with each population characterized by specific allele frequencies. However, despite their widespread use, there's been a significant lack of attention paid to how well these models actually fit the data they are applied to. The authors argue that without assessing model fit, researchers might draw incorrect conclusions about population history or mistakenly correct for population structure, leading to spurious genetic associations. This paper aims to fill that gap by providing a systematic way to quantify the goodness-of-fit of admixture models.

Methods

The core of this paper is the application of Posterior Predictive Checks (PPCs), a Bayesian technique for model assessment. The general PPC workflow involves:

  1. Fitting the model: An admixture model is fitted to the observed genetic data to estimate its parameters (e.g., ancestral population proportions, allele frequencies).
  2. Generating synthetic data: Using the fitted model, numerous "replicated" datasets are generated. These synthetic datasets share the same latent population structure as the observed data.
  3. Comparing statistics: A "discrepancy function" is used to summarize a specific aspect of the data. This function is applied to both the observed data and the replicated datasets.
  4. Assessing fit: The observed discrepancy is compared to the distribution of discrepancies from the replicated data. If the observed discrepancy falls outside the typical range of replicated discrepancies, it suggests the model is a poor fit for that specific aspect of the data.

The authors developed five novel discrepancy functions tailored to common assumptions and applications of admixture models:

  • Identity by State (IBS): Measures within-population genetic variation, testing if the model accurately captures similarity among individuals descended from the same ancestral population.
  • Background Linkage Disequilibrium (LD): Assesses the model's assumption of SNP independence by quantifying short-range correlations between SNPs.
  • FST: Evaluates the appropriateness of the chosen number of ancestral populations by comparing allele frequency variance between inferred and reported ancestries.
  • Assignment Uncertainty (Entropy): Quantifies how uncertain the model is about assigning alleles to specific ancestral populations, probing the distinctiveness of these populations.
  • Association Mapping Correction: Tests whether the inferred population structure effectively controls for confounding in genome-wide association studies.

Datasets

The authors evaluated their PPC framework on four publicly available genomic datasets:

  • HapMap phase 3: Individuals clustered by geographic origin, fitted with 6 ancestral populations.
  • POPRES (Population Reference Sample): European individuals, fitted with 4 ancestral populations.
  • African American (ASW): Individuals with reported African ancestry, fitted with 2 ancestral populations.
  • Continental Indians: Individuals from tribal groups in India, fitted with 2 ancestral populations.

These datasets were chosen to represent qualitatively different types of latent population structure.

Key Insights and Ideas

  • Model fit is study-specific: The same admixture model can exhibit vastly different levels of misspecification depending on the dataset. This underscores that a "one-size-fits-all" approach to model fitting is insufficient.
  • PPCs are crucial for interpretation: The authors emphasize that without PPCs, researchers might be misled by model assumptions, leading to incorrect inferences about population history or spurious genetic associations.
  • Discrepancy functions reveal specific model weaknesses: Each of the five developed discrepancy functions targets a different assumption of the admixture model, allowing researchers to pinpoint why a model might be failing.
  • LD is a pervasive challenge: The background LD discrepancy function frequently indicated model misspecification, suggesting that the assumption of SNP independence is often violated and can lead to phantom ancestral populations.
  • FST is a useful tool for selecting K: The FST discrepancy function proved effective in evaluating the appropriate number of ancestral populations (K) for a given dataset.

Key Results

  • The PPCs revealed that the POPRES dataset showed consistent underestimation of average inter-individual similarity, indicating model misspecification for within-population variation.
  • The background LD discrepancy often showed misspecification, particularly at small lags, across all studies, highlighting the impact of the independence assumption.
  • The FST discrepancy suggested that for the ASW study, the chosen number of ancestral populations (K=2) might be insufficient, potentially due to LD.
  • The entropy discrepancy indicated gross misspecification for the ASW study regarding allele-specific ancestry assignments.
  • The association mapping correction PPC generally showed good performance, providing confidence in the model's ability to reject false positives, though the POPRES data showed some deviation.

Limitations and Future Work

  • LD modeling: The authors frequently suggest that explicitly modeling linkage disequilibrium (LD) would improve admixture model fit, particularly in datasets where LD is prevalent.
  • Continuous population models: For datasets with continuous geographic gradients in ancestry (like POPRES), the authors propose exploring continuous population models (e.g., principal component-based) instead of discrete ancestral populations.
  • Selecting K: While the FST discrepancy helps, selecting the optimal number of ancestral populations (K) remains a significant challenge in admixture modeling.
  • Generalizability of PPCs: The authors emphasize that the PPC methodology is broadly applicable to any probabilistic model and could be valuable for evaluating demographic models and other complex statistical analyses in genomics. They advocate for an iterative model-fitting and checking process.

Summary updated

Abstract

Admixture models are a ubiquitous approach to capture latent population structure in genetic samples. Despite the widespread application of admixture models, little thought has been devoted to the quality of the model fit or the accuracy of the estimates of parameters of interest for a particular study. Here we develop methods for validating admixture models based on posterior predictive checks (PPCs), a Bayesian method for assessing the quality of fit of a statistical model to a specific dataset. We develop PPCs for five population-level statistics of interest: within-population genetic variation, background linkage disequilibrium, number of ancestral populations, between-population genetic variation, and the downstream use of admixture parameters to correct for population structure in association studies. Using PPCs, we evaluate the quality of the admixture model fit to four qualitatively different population genetic datasets: the population reference sample (POPRES) European individuals, the HapMap phase 3 individuals, continental Indians, and African American individuals. We found that the same model fitted to different genomic studies resulted in highly study-specific results when evaluated using PPCs, illustrating the utility of PPCs for model-based analyses in large genomic studies.
BibTeX
@inproceedings{mimno2015posterior,
  author = {David Mimno and David M. Blei and Barbara E. Engelhardt},
  title = {{Posterior predictive checks to quantify lack-of-fit in admixture models of latent population structure}},
  booktitle = {Proceedings of the National Academy of Sciences},
  year = {2015},
  doi = {10.1073/pnas.1412301112},
}
Figure