Profiling the baseline performance and limits of machine learning models for adaptive immune receptor repertoire classification

Chakravarthi Kanduri, Milena Pavlović, Lonneke Scheffer, Keshav Motwani, Maria Chernigovskaya, Victor Greiff, Geir Kjetil SandveGigaScience 2022 · DOI

TL;DR

This paper investigates the performance limits of basic machine learning (ML) models for classifying immune states based on adaptive immune receptor repertoires (AIRRs). By creating a large, synthetic benchmark dataset, they demonstrate that even simple models like L1-penalized logistic regression can achieve high accuracy in identifying subtle immune signals, but their performance degrades under increased signal complexity. This work provides crucial insights for developing more sophisticated ML methods by highlighting scenarios where current approaches are sufficient and where new methods are truly needed.

What Problem Does This Paper Address?

The adaptive immune system's ability to recognize a vast array of threats relies on a diverse repertoire of immune receptors (AIRs). Sequencing these repertoires (AIRRs) offers a snapshot of an individual's immune state, potentially revealing past or present responses to diseases like infections or cancer. Machine learning (ML) has emerged as a powerful tool for analyzing AIRR data to predict these immune states. However, the field has seen a surge in complex ML models without a clear understanding of when simpler, established methods might suffice. This lack of a systematic evaluation of "baseline" performance hinders the development of truly novel ML approaches by not clearly defining the frontiers where advanced methods are genuinely required. The authors aim to address this by profiling the capabilities and limitations of basic ML models across a wide range of AIRR data and immune signal complexities.

Methods

The core of this study involves generating a comprehensive suite of synthetic AIRR benchmark datasets. These datasets were created using the OLGA simulation suite, which models human V(D)J recombination to produce "native-like" TCRβ CDR3 sequences. The authors then systematically varied several key properties:

  • Dataset Properties: Sample size (number of repertoires), repertoire size (number of sequences per repertoire), class balance, and noise in the negative class.
  • Immune Signal Properties: Witness rate (how often a signal appears in positive examples), number and size of signal motifs (k-mers), and whether motifs are continuous or gapped.
  • ML Model Assumptions: They primarily used L1-penalized logistic regression, a standard "sparse" model, and explored variations in regularization strength. They also briefly compared it with Support Vector Classifiers (SVC) and Random Forests (RF).

The performance was evaluated using balanced accuracy via nested cross-validation, with results replicated three times for robustness.

Datasets

The study did not use experimental datasets for training or primary evaluation. Instead, they generated a large collection of synthetic AIRR benchmark datasets. These synthetic datasets were based on:

  • OLGA-based repertoires: Generated using a VDJ recombination model to mimic native-like TCRβ CDR3 sequences.
  • Random repertoires: Composed of random amino acid sequences.
  • Meta repertoires: Repertoires composed of randomly pooled experimental sequences.
  • Experimental repertoires: Subsampled whole experimental repertoires.

The primary focus was on OLGA-based repertoires for constructing the benchmark datasets, totaling approximately 1,000 datasets with around 250,000 repertoires and ~46 billion TCRβ CDR3 sequences.

Key Insights and Ideas

  • Simplicity Can Be Powerful: Even a basic L1-penalized logistic regression model can achieve high prediction accuracy in classifying immune states, even when the immune signal is extremely rare (as low as 2 in 100,000 sequences). This challenges the assumption that highly complex ML models are always necessary.
  • Signal Complexity is the Bottleneck: The performance of baseline models degrades significantly as the complexity of the immune signal increases. This includes scenarios with multiple, gapped, or less frequent motifs.
  • Matching Model Assumptions to Data is Crucial: The performance of ML models is highly sensitive to how well their assumptions about the "ground truth" signal (e.g., k-mer size, presence of gaps) align with the actual signal in the data. Mismatches lead to substantial performance drops.
  • Data Characteristics Matter: Factors like sample size, repertoire size, and class balance all influence model performance, particularly at lower witness rates. Increasing repertoire size generally leads to better performance.
  • Distributional Shift is a Real Challenge: Models trained on data with one signal distribution perform less well when applied to data with a different distribution, especially if the test data has a lower signal rate. However, models can sometimes adapt better if the test data has a higher signal rate than the training data.

Key Results

  • High Accuracy with Rare Signals: L1-penalized logistic regression achieved high prediction accuracy (e.g., 99% accuracy at a 0.005% witness rate with a single motif) even at very low signal frequencies.
  • Performance Degradation: With 64 implanted motifs, the baseline model struggled at lower witness rates, indicating that increased signal complexity significantly hinders performance.
  • Sample Size Impact: A moderate increase in sample size (number of repertoires) was not sufficient for substantial performance gains, especially at low witness rates. However, increasing repertoire size (number of sequences per repertoire) did show performance improvements.
  • Class Balance and Distributional Shift: Class imbalance had a moderate impact, but distributional shifts (changes in signal rate between training and test sets) significantly affected performance, with test set shifts being more impactful.
  • Model vs. Data Assumptions: Models performed best when their assumed feature size matched the ground truth signal size. Mismatched assumptions led to poorer performance.
  • Noise Impact: Increasing noise in the negative class examples reduced prediction accuracy, but the model remained effective even when noise levels were half the witness rate.

Limitations and Future Work

  • Synthetic Data Reliance: The study relies heavily on synthetic data. While designed to mimic real-world complexities, it may not fully capture all nuances of experimental AIRR data (e.g., sequencing artifacts, batch effects). Future work should explore these additional noise sources.
  • Focus on TCRβ CDR3: Experiments were primarily conducted on TCRβ CDR3 sequences. While principles may apply broadly, the impact of BCR somatic hypermutation warrants further investigation.
  • Limited ML Model Scope: The study focused on baseline models, particularly L1-penalized logistic regression. While they briefly explored RF and SVC, a comprehensive benchmarking of state-of-the-art AIRR-ML methods is still needed.
  • Future Benchmarking: The authors propose their generated benchmark dataset as a template for future, more specialized benchmarking efforts, especially for identifying scenarios where novel, sophisticated ML methodologies are truly required.
  • MIL for Complex Signals: The study suggests that Multiple Instance Learning (MIL) approaches might be beneficial for scenarios with a large number of phenotype-associated patterns, which current baseline models struggle with.
  • Trade-off between Sample Size and Sequencing Depth: Further research is needed to understand the optimal balance between sample size and sequencing depth for effective AIRR-ML.

Summary updated

Abstract

We provide a reference benchmark to guide new AIRR-ML classification methodology by (i) identifying those scenarios characterized by immune signal and data set complexity, where baseline methods already achieve high prediction accuracy, and (ii) facilitating realistic expectations of the performance of AIRR-ML models given training data set properties and assumptions. Our study serves as a template for defining specialized AIRR benchmark data sets for comprehensive benchmarking of AIRR-ML methods.
BibTeX
@inproceedings{kanduri2022profiling,
  author = {Chakravarthi Kanduri and Milena Pavlović and Lonneke Scheffer and Keshav Motwani and Maria Chernigovskaya and Victor Greiff and Geir Kjetil Sandve},
  title = {{Profiling the baseline performance and limits of machine learning models for adaptive immune receptor repertoire classification}},
  booktitle = {GigaScience},
  year = {2022},
  doi = {10.1093/gigascience/giac046},
}
Figure