Benchmarking Machine-Learning Classifiers for Neoantigen Immunogenicity Prediction and the Limits of SHAP Interpretability

Ramon GonzalezarXiv 2026 · arXiv

TL;DR

This paper benchmarks 60 machine-learning configurations for predicting neoantigen immunogenicity, revealing that classical linear models surprisingly outperform modern complex ensembles. Crucially, it exposes a "SHAP trap" where standard interpretability tools can confidently assign high importance to meaningless, high-cardinality features, and proposes a simple "shuffled-label control" to prevent these misleading conclusions.

What Problem Does This Paper Address?

Personalized cancer vaccines (PCVs) aim to realize Paul Ehrlich’s 1908 vision of a "magic bullet": a treatment that destroys foreign tumor cells while leaving healthy tissue untouched. The modern bottleneck isn't building the bullet, but choosing the target.

A single tumor can yield thousands of mutant peptides (neoantigens), but only a tiny fraction will actually trigger a T-cell response (immunogenicity). Computational "candidate selection" is the filter used to find these needles in the haystack. However, current state-of-the-art predictors often perform barely better than chance on experimentally validated data. This study investigates whether general-purpose machine learning can do better and, more importantly, whether we can trust the "explanations" these models give us for their predictions.

Biological context of personalized cancer vaccination and the scope of this work.

Methods

The authors implemented a rigorous machine-learning framework (shown in Figure 2) consisting of three main pillars:

  1. A Massive Benchmark: They compared 12 supervised classifier families (ranging from Logistic Regression to XGBoost and Neural Networks), each in five different configurations, totaling 60 models.
  2. Rigorous Validation: They used a stratified nested k-fold cross-validation protocol. This ensures that the data used to tune a model's hyperparameters is never the same data used to evaluate its final performance—a common pitfall that leads to over-optimistic results.
  3. Interpretability Stress-Test: They combined SHAP (SHapley Additive exPlanations) with feature ablation (removing a feature and retraining to see if performance drops). To verify SHAP values, they introduced a shuffled-label control: they randomly permuted categorical labels to see if SHAP would still "credit" them with importance even when they contained zero information.
The implemented machine-learning framework for neoantigen immunogenicity prediction.

Datasets

The study utilized the Immunogenic Tumor Specific Neoantigen database (ITSNdb).

  • Size: 199 experimentally validated neoantigens (129 immunogenic, 70 non-immunogenic).
  • Significance: Unlike larger datasets that mix predicted and validated data, ITSNdb only includes peptides with experimentally confirmed MHC binding and immunogenicity readouts. This makes it a "harder" but more realistic benchmark.

Key Insights and Ideas

  • The Complexity Paradox: In an era of deep learning, the "simpler" models won. Linear Discriminant Analysis (LDA) and Logistic Regression (LR) outperformed gradient-boosted trees and neural networks. The authors suggest that with small datasets (like the 199 peptides here), complex models simply don't have enough signal to justify their high capacity.
  • The SHAP Trap: SHAP is often treated as the "ground truth" for model interpretability. However, this paper shows that for high-cardinality categorical features (like GeneSymbol, which had 173 unique values for 199 peptides), SHAP can fail spectacularly. It assigned up to 81.7% of its importance value to the gene source, even though feature ablation proved the gene was actually dispensable.
  • The "Earned" SHAP Value: By shuffling the gene labels, the authors found that the meaningless, randomized column earned the same high SHAP value as the real one. This proves that SHAP was merely measuring the "cardinality" (the number of unique categories) rather than actual biological signal.

Key Results

  • Top Performers: LDA achieved the highest balanced accuracy (0.760 ± 0.100), while SGD (Stochastic Gradient Descent) led in ranking performance with a ROC-AUC of 0.817 ± 0.100.
  • Feature Importance: Feature ablation established the HLA allele as the dominant predictive input. Removing it caused performance to drop across all 12 model families.
  • Performance Gap: The study highlights a significant drop in performance when tools are moved from "simulated" datasets to experimentally validated ones. As seen in Table 1, tools reporting ROC-AUCs of ~0.85 on other datasets fell to 0.52–0.60 when re-evaluated on ITSNdb.
Table 1: Machine-learning approaches to neoantigen immunogenicity prediction: data, methods and reported performance. The values in the last column are not comparable across rows, because each was obtained on a different dataset with a different protocol. The contrast between rows 3 and 6 illustrates the point: tools reporting ROC-AUC near 0.85 on IEDB-derived data fall to 0.52 to 0.60 when re-evaluated on experimentally validated neoantigens with both labels. Values are as reported by the authors of each study.

Limitations and Future Work

  • Sample Size: The primary limitation is the small cohort of 199 peptides. While this reflects the scarcity of high-quality validated data, it limits the ability of complex models to learn.
  • Scaling Up: The authors suggest re-running this protocol on larger platforms like TumorAgDB (1,106 neoantigens) to see if the advantage of linear models holds as data volume increases.
  • Redundancy Analysis: Future work could involve more granular feature selection (e.g., mutual information or wrapper methods) to resolve redundancies between features like tumor type and HLA alleles.

The takeaway for researchers? Before trusting a SHAP plot for your high-cardinality data, run a shuffled-label control. You might find your model is "learning" from nothing but the noise of its own encoding.

Summary updated

Abstract

Personalized cancer vaccines require prioritizing, among hundreds of candidate mutant peptides, the few that will induce a T-cell response. This study asks what general-purpose machine learning achieves on that task, and whether the features such models identify as important can be trusted. We make three contributions. The first is a benchmark of sixty models: twelve supervised classifier families, each in five configurations, evaluated on the Immunogenic Tumor Specific Neoantigen database (ITSNdb; 199 experimentally validated neoantigens, 129 immunogenic and 70 non-immunogenic). All sixty are compared under one stratified k-fold cross-validation protocol, in which the data used to choose a configuration are never the data used to score it. The second is an interpretability study that combines SHAP with feature ablation. The third is a shuffled-label control that tests whether a SHAP value has been earned. Two results follow, and both run against current practice. First, the classical linear models lead the ranking. The gradient-boosted ensembles and neural networks that dominate recent work bring no advantage. Second, SHAP can fail on high-cardinality categorical features. Ablation establishes the presenting HLA allele as the dominant input and the source gene as dispensable. SHAP reports the opposite, assigning 68.7% to 81.7% of its total value to the source gene in every model that gives each gene its own weight. Shuffling the gene labels at random among the peptides resolves the contradiction: the meaningless column earns the same SHAP value as the real one. That value measures only that ITSNdb contains 173 genes for 199 peptides. We therefore recommend that SHAP values for high-cardinality categorical annotations in cohorts of this size always be reported against such a shuffled-label control, and we describe one in full.
BibTeX
@misc{gonzalez2026benchmarking,
  author = {Ramon Gonzalez},
  title = {{Benchmarking Machine-Learning Classifiers for Neoantigen Immunogenicity Prediction and the Limits of SHAP Interpretability}},
  year = {2026},
  eprint = {2611.00907},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.00907},
}
Figure