Comparing multi- and single-sample variant calls to improve variant call sets from deep coverage whole-genome sequencing data

Suyash Shringarpure, Rasika A. Mathias, Ryan D. Hernandez, Timothy D. O’Connor, Zachary A. Szpiech, Raúl Torres, Francisco M. De La Vega, Carlos D. Bustamante, Kathleen C. Barnes, Margaret A. TaubbioRxiv 2016 · DOI

TL;DR

This paper introduces a novel method using genotype array data to train a Random Forest classifier that accurately distinguishes true genetic variants from false positives in whole-genome sequencing data. This approach offers a more robust and dataset-specific quality control for variant calls compared to generic methods, improving the reliability of genomic analyses.

What Problem Does This Paper Address?

Next-generation sequencing (NGS) technologies have revolutionized our ability to study genetic variation. However, the process of "variant calling" – identifying differences in DNA sequences – is prone to errors, leading to both false positives (calling a variant that isn't there) and false negatives (missing a true variant). These errors can significantly impact downstream analyses, especially in large-scale population genetics studies. A key decision in variant calling is whether to use single-sample callers (analyzing each genome independently) or multi-sample callers (leveraging information across multiple genomes). While multi-sample callers theoretically offer advantages by borrowing information, it's not always clear which approach is best, and the quality of calls from any method can be difficult to assess reliably. Existing methods for assessing variant quality often rely on public databases like HapMap or dbSNP, which may not perfectly represent the specific population being studied or the nuances of the sequencing data itself.

Methods

The core of this paper is a Random Forest classifier trained to assess the quality of variant calls. Instead of relying on generic databases, the authors use genotype array data (specifically, Illumina HumanOmni2.5 BeadChip data) from the same individuals whose whole-genome sequencing data was analyzed. This array data serves as a "ground truth" for variant presence.

Here's a breakdown of their approach:

  • Callset Generation: They generated variant calls using three different algorithms:
  • Illumina CASAVA: A single-sample caller.
  • Real Time Genomics (RTG): A multi-sample caller.
  • GATK Unified Genotyper: Another multi-sample caller.
  • Classifier Training: For each callset, they trained a separate Random Forest classifier. The input features for the classifier included various genomic characteristics of each variant (e.g., quality scores, depth of coverage, allele balance, GC content, mapping quality). The "true" label for training was determined by whether a variant identified by the sequencing data was also present in the genotype array data for that individual.
  • Orthogonal Validation: To further validate their method, they used a technical replicate of one sample. This allowed them to check if variants found in the original callset were also found in the replicate, providing an independent measure of reliability.
  • Handling Imbalanced Data: A significant challenge was the highly unbalanced nature of the training data (many more sites labeled as "variant" than "non-variant" based on array data). They addressed this using the SMOTE (Synthetic Minority Over-sampling Technique) method, which synthetically creates examples of the minority class to balance the dataset.

Datasets

  • Whole-Genome Sequencing Data: 642 African-ancestry genomes from the Consortium on Asthma among African-ancestry Populations in the Americas (CAAPA), sequenced to high depth (~30X).
  • Genotype Array Data: Illumina HumanOmni2.5 BeadChip data for the same 642 individuals. This served as the "ground truth" for training and validation.
  • Technical Replicate: A technical replicate of one sample was used for orthogonal validation.

The paper does not introduce a new benchmark dataset but rather leverages existing genotype array data as a novel, dataset-specific validation resource.

Key Insights and Ideas

  • Genotype Arrays as Ground Truth: The most significant insight is the utility of readily available genotype array data as a highly accurate, sample-specific ground truth for validating NGS variant calls. This bypasses the limitations of relying on generic, potentially population-biased public databases.
  • Callset-Specific Classifiers: Training separate classifiers for each variant calling algorithm (Illumina, RTG, GATK) proved more effective than a single generic classifier. This acknowledges that different algorithms have distinct strengths and weaknesses, and their variant calls have different characteristics.
  • Allele Frequency Matters: The frequency of a variant within the dataset was a crucial feature for distinguishing true from false positives, highlighting that rare variants might require different quality assessment criteria than common ones.
  • Imbalanced Data is Manageable: The successful application of SMOTE demonstrates that even with highly imbalanced training data (where true variants are rare compared to non-variants), robust classifiers can be built.

Key Results

  • High Accuracy: The callset-specific Random Forest classifiers achieved very high accuracy. For example, at a 5% False Positive Rate (FPR), the True Positive Rates (TPR) were 97.5% for CASAVA, 95% for RTG, and 99% for GATK.
  • Feature Importance: The study identified key features that contribute to variant quality, with allele frequency, variant quality score (QUAL), and depth of coverage being particularly important.
  • Callset Differences: The analysis revealed differences in the overlap and characteristics of variants called by single-sample vs. multi-sample callers. For instance, the Illumina single-sample callset had the largest number of sites predicted to be variable, while GATK had the highest proportion of sites predicted to be variable.
  • Orthogonal Validation Success: Variants found in both replicates consistently received higher classifier scores, supporting the method's ability to identify truly variable sites.

Limitations and Future Work

  • False Positive Rate: The authors acknowledge that a 5% FPR, while good for this method, might still be considered high for applications in disease genetics, emphasizing the continued need for replication or further genotyping of interesting findings.
  • Future Directions:
    • Consensus Call Sets: They suggest developing a method to produce a combined, consensus call set by weighting the outputs of different callers based on their classifier scores or using a latent variable model.
  • Allele Frequency-Specific Tuning: The method could be further refined by adjusting quality thresholds or retraining classifiers for specific allele frequency bins, especially for variants of particular interest.
  • Broader Applicability: While demonstrated on African-ancestry genomes, the method's reliance on genotype array data makes it adaptable to other populations and datasets, provided such array data is available.

Summary updated

Abstract

ABSTRACT Motivation Variant calling from next-generation sequencing (NGS) data is susceptible to false positive calls due to sequencing, mapping and other errors. To better distinguish true from false positive calls, we present a method that uses genotype array data from the sequenced samples, rather than public data such as HapMap or dbSNP, to train an accurate classifier using Random Forests. We demonstrate our method on a set of variant calls obtained from 642 African-ancestry genomes from the The Consortium on Asthma among African-ancestry Populations in the Americas (CAAPA), sequenced to high depth (30X). Results We have applied our classifier to compare call sets generated with different calling methods, including both single-sample and multi-sample callers. At a False Positive Rate of 5%, our method determines true positive rates of 97.5%, 95% and 99% on variant calls obtained using Illumina’s single-sample caller CASAVA, Real Time Genomics’ multisample variant caller, and the GATK Unified Genotyper, respectively. Since most NGS sequencing data is accompanied by genotype data for the same samples, our method can be trained on each dataset to provide a more accurate computational validation of site calls compared to generic methods. Moreover, our method allows for adjustment based on allele frequency (e.g., a different set of criteria to determine quality for rare vs. common variants) and thereby provides insight into sequencing characteristics that indicate data quality for variants of different frequencies. Availability Code will be made available prior to publication on Github.
BibTeX
@misc{shringarpure2016comparing,
  author = {Suyash Shringarpure and Rasika A. Mathias and Ryan D. Hernandez and Timothy D. O’Connor and Zachary A. Szpiech and Raúl Torres and Francisco M. De La Vega and Carlos D. Bustamante and Kathleen C. Barnes and Margaret A. Taub},
  title = {{Comparing multi- and single-sample variant calls to improve variant call sets from deep coverage whole-genome sequencing data}},
  year = {2016},
  doi = {10.1101/078642},
  howpublished = {bioRxiv},
}
Figure