Identification and validation of 174 COVID-19 vaccine candidate epitopes reveals low performance of common epitope prediction tools

Marek Prachař, Sune Justesen, Daniel Bisgaard Steen-Jensen, Stephan Thorgrimsen, Erik Jurgons, Ole Winther, Frederik Otzen BaggerScientific 2020 · DOI

TL;DR

This paper benchmarks 19 popular epitope prediction tools against a novel peptide-MHC stability assay for SARS-CoV-2. They found that most tools exhibit high false positive rates, meaning many predicted binders are unstable and unlikely to elicit an immune response, posing a significant challenge for vaccine design. The authors provide a validated list of 174 SARS-CoV-2 epitopes that do bind stably to 11 different HLA alleles.

What Problem Does This Paper Address?

Developing a vaccine against a novel pathogen like SARS-CoV-2 requires rapidly identifying potential targets that can trigger a protective immune response. A key step in this process is identifying "epitopes" – specific parts of viral proteins that T-cells recognize. T-cells recognize these epitopes when they are presented by Major Histocompatibility Complex (MHC) molecules on the surface of infected cells. However, experimentally testing every possible peptide from a large virus like SARS-CoV-2 is computationally infeasible.

To overcome this, researchers rely on computational tools that predict which peptides are likely to bind to specific MHC molecules (HLA alleles in humans). These predictions are crucial for narrowing down the candidates for experimental validation. The problem this paper addresses is the reliability of these prediction tools. Specifically, it questions whether current prediction tools are accurate enough to confidently identify stable peptide-MHC binders that are likely to elicit a meaningful immune response, especially in the context of vaccine development where false positives can lead to wasted effort and potentially ineffective vaccines.

Methods

The core of this study involved a two-pronged approach:

  1. Epitope Prediction: The researchers used a suite of 19 well-established and newer epitope prediction tools (listed in Table 1) to identify potential SARS-CoV-2 epitopes. They focused on predicting peptides that would bind to 11 different HLA class I alleles and 1 HLA class II allele, covering a broad range of human populations.
  2. Peptide-MHC Stability Assay (NeoScreen): To validate the predictions, they employed a novel in vitro assay called NeoScreen. This assay measures the stability of the peptide-MHC complex under denaturing conditions (using urea). A stable complex is considered more likely to be presented to T-cells and thus more relevant for vaccine design. They assessed 777 unique peptides predicted by the tools.

They also developed a proof-of-concept prediction model (PrdX 1.0) for a specific HLA allele (HLA-A*02:01) using their own in-house stability data to see if improved training data could enhance prediction accuracy.

Datasets

  • SARS-CoV-2 Genome: The reference sequence of SARS-CoV-2 (Accession MN908947.3) was used to generate potential peptide sequences.
  • Publicly Available Training Data: The paper mentions that traditional prediction tools are trained on data from affinity assays and that the Immune Epitope Database (IEDB) is a major source of such data.
  • In-house Stability Data: For their PrdX 1.0 model, the researchers utilized 2193 historical in-house stability data points for the HLA-A*02:01 allele. This data included a mix of human cancer-derived and random peptide measurements.
  • Validated Epitopes: The paper makes freely available a list of 174 validated SARS-CoV-2 epitopes that bind stably to 11 HLA alleles. This dataset is intended to aid vaccine design.

Key Insights and Ideas

  • Prediction Tools Fall Short: The most striking insight is the significant discrepancy between what prediction tools forecast and what is experimentally validated as stable peptide-MHC binding. This highlights a critical limitation in current computational approaches for vaccine candidate identification.
  • Stability Matters More Than Affinity: The study emphasizes that while affinity (how strongly a peptide binds) is important, the stability of the peptide-MHC complex is a more crucial indicator for T-cell recognition and thus vaccine efficacy. Many high-affinity binders may not be stable enough to be presented effectively.
  • HLA Allele Dependence is High: The performance of prediction tools is not uniform; it varies significantly depending on the specific HLA allele being considered. This suggests that models trained on one allele may not generalize well to others.
  • Training Data Quality is Key: The poor performance of tools, especially for the well-studied HLA-A*02:01 allele, suggests that the quality and relevance of publicly available training data might be a bottleneck. Their custom-trained model (PrdX 1.0) outperformed others on this allele, supporting this hypothesis.
  • Cross-Reactivity Potential: The overlap of 98 validated epitopes with those previously deposited in the IEDB, particularly those from SARS-CoV, suggests significant cross-reactivity. This has implications for understanding pre-existing immunity and designing vaccines that could leverage this.

Key Results

  • High False Positive Rate: The study found a high false positive rate across all tested prediction tools when assessing peptide-MHC stability.
  • 174 Validated Epitopes: The researchers identified and validated 174 SARS-CoV-2 epitopes that bind stably to 11 HLA alleles. Of these, 73 are novel, while 98 have been previously reported.
  • Tool Performance Varies: No single tool consistently outperformed others across all HLA alleles. NetMHC 4.0, NetMHCcons 1.1, NetMHCpan_BA 4.0, and IEDB-AR Consensus showed strong performance for some alleles, but overall, performance was allele-dependent.
  • PrdX 1.0 Outperforms: Their custom-trained model, PrdX 1.0, showed improved performance for HLA-A*02:01 compared to other tested tools.
  • Spearman Correlation: Spearman correlation analysis revealed significant inconsistencies in tool performance depending on the predicted allele, with some tools like PSSMHCpan 1.0 showing higher consistency but lower median correlation.

Limitations and Future Work

  • Limited HLA Class II Coverage: The study focused primarily on HLA class I alleles, with only one HLA class II allele tested.
  • NeoScreen Assay Limitations: While novel, the NeoScreen assay's performance is compared to other methods, and its broad applicability across all HLA types might require further validation.
  • Training Data Hypothesis: The hypothesis that training data quality is the primary limitation for tool performance is supported by their PrdX 1.0 results but could be further investigated with more extensive data curation and model development.
  • Future Work: The authors suggest that their validated list of epitopes can aid in the design of efficacious SARS-CoV-2 vaccines. Further research could focus on improving the accuracy of prediction tools by incorporating more diverse and higher-quality training data, exploring more advanced machine learning architectures, and expanding the validation to a wider range of HLA alleles and experimental readouts (e.g., T-cell activation assays).

Summary updated

Abstract

The outbreak of SARS-CoV-2 (2019-nCoV) virus has highlighted the need for fast and efficacious vaccine development. Stimulation of a proper immune response that leads to protection is highly dependent on presentation of epitopes to circulating T-cells via the HLA complex. SARS-CoV-2 is a large RNA virus and testing of all of its overlapping peptides in vitro to deconvolute an immune response is not feasible. Therefore HLA-binding prediction tools are often used to narrow down the number of peptides to test. We tested NetMHC suite tools' predictions by using an in vitro peptide-MHC stability assay. We assessed 777 peptides that were predicted to be good binders across 11 MHC alleles in a complex-stability assay and tested a selection of 19 epitope-HLA-binding prediction tools against the assay. In this investigation of potential SARS-CoV-2 epitopes we found that current prediction tools vary in performance when assessing binding stability, and they are highly dependent on the MHC allele in question. Designing a COVID-19 vaccine where only a few epitope targets are included is therefore a very challenging task. Here, we present 174 SARS-CoV-2 epitopes with high prediction binding scores, validated to bind stably to 11 HLA alleles. Our findings may contribute to the design of an efficacious vaccine against COVID-19.
BibTeX
@inproceedings{prachar2020identification,
  author = {Marek Prachař and Sune Justesen and Daniel Bisgaard Steen-Jensen and Stephan Thorgrimsen and Erik Jurgons and Ole Winther and Frederik Otzen Bagger},
  title = {{Identification and validation of 174 COVID-19 vaccine candidate epitopes reveals low performance of common epitope prediction tools}},
  booktitle = {Scientific},
  year = {2020},
  doi = {10.1038/s41598-020-77466-4},
}
Figure