Revealing the hidden sequence distribution of epitope-specific TCR repertoires and its influence on machine learning model performance

Sofie Gielis, Maria Chernigovskaya, Milena Pavlović, Vincent Van Deuren, Romi Vandoren, Sebastiaan Valkiers, Kris Laukens, Victor Greiff, Pieter MeysmanbioRxiv 2024 · DOI

TL;DR

This paper investigates how the natural diversity of T-cell receptor (TCR) sequences that recognize the same epitope impacts the performance of machine learning models designed to predict TCR-epitope interactions. They found that the composition of training data, specifically the distribution of TCR sequences and their similarity to "negative" (non-binding) sequences, significantly influences model accuracy, suggesting that careful data curation is crucial for building better predictive models.

What Problem Does This Paper Address?

Predicting which T-cell receptors (TCRs) will bind to specific epitopes is a fundamental challenge in immunology, with implications for vaccine development, cancer immunotherapy, and understanding autoimmune diseases. Machine learning models have shown promise in this area, but their performance is heavily reliant on the quality and composition of the training data. While many studies focus on developing new prediction algorithms, this paper highlights a critical gap: the lack of attention paid to the natural distribution of TCR sequences within epitope-specific repertoires and how this distribution might inherently bias or limit model performance. Specifically, they question how similar TCRs need to be to recognize the same epitope and how diverse these sequences can naturally be, and how these characteristics affect model predictability.

Methods

The researchers employed a multi-pronged approach:

  • Clustering: They used a Hamming distance of one on amino acid sequences to cluster TCR beta (β) chains from epitope-specific repertoires. This allowed them to identify groups of similar TCRs that recognize the same epitope. They also used a Markov Cluster (MCL) approach for clustering.
  • Motif Generation: Consensus motifs were generated from these clusters to represent the general TCR signatures within epitope-specific repertoires.
  • Simulation: Synthetic TCR repertoires were generated in silico using the LIgO AIRR simulation software suite. This allowed them to control the input motifs and study the origin of sequence diversity under specific constraints.
  • Similarity Metrics: They calculated Levenshtein distances to quantify the similarity between TCR sequences, both within epitope-specific repertoires and between epitope-specific and negative training data.
  • Performance Evaluation: The TCRex prediction model was used as a baseline to evaluate how different data characteristics (e.g., cluster capacity, diversity, overlap with negative data) correlated with model performance metrics like AUROC (Area Under the Receiver Operating Characteristic curve) and average precision.

Datasets

  • TCRex Model Training Data: This was the primary dataset used, comprising epitope-specific TCR β chain data for 6 cancer and 120 viral epitopes. It was compiled from public databases like VDJdb, McPAS-TCR, and the ImmuneCODE™ database.
  • Simulated TCR Repertoires: Generated using LIgO, these datasets served to investigate the origin of sequence distribution and test hypotheses about TCR generation probabilities.
  • Negative Training Data: Collected from non-stimulated TCR repertoires of healthy individuals, this data was crucial for evaluating model performance and understanding overlap.

Key Insights and Ideas

  • Diversity is Key, but Not Always Beneficial: While epitope-specific TCR repertoires exhibit significant diversity, this diversity isn't always conducive to better machine learning model performance. Clonal repertoires (few large clusters) tend to lead to more performant models.
  • Motifs Drive Diversity: The observed diversity in epitope-specific TCR repertoires can arise from a relatively small set of core binding motifs, constrained by the natural VDJ recombination process.
  • Negative Data Matters: The similarity between epitope-specific TCRs and "negative" (non-binding) TCRs in the training data significantly impacts model performance. High overlap leads to poorer predictions.
  • Data Composition is Crucial: The paper strongly emphasizes that the composition of the training data—both positive and negative—is as important as the prediction algorithm itself for achieving accurate TCR-epitope binding predictions.

Key Results

  • Model Performance Correlates with Clustering: A clear trend was observed where a higher percentage of clustered TCRs within the epitope-specific training data correlated with better model performance (AUROC).
  • Clonal Repertoires Outperform Diverse Ones: Models trained on more "clonal" TCR repertoires (fewer, larger clusters) generally performed better than those trained on highly diverse repertoires. This was supported by metrics like Pielou's index, Gini coefficient, and DE50.
  • Overlap with Negative Data Hinders Performance: Models trained on data with high similarity between epitope-specific and negative TCR sequences showed poorer AUROC performance.
  • TCRs Can Be Similar to Negative Sequences: The study found that non-epitope-specific TCRs can be very similar to epitope-specific TCRs, sometimes differing by only one amino acid. This similarity can blur the lines for predictive models.

Limitations and Future Work

  • Limited Data for Some Epitopes: The study acknowledges that for some epitopes, the number of available TCR sequences is limited, which can affect the reliability of conclusions drawn from those specific datasets.
  • Unknown TCR Generation Model: For experimental data, the exact TCR generation model for individuals is unknown, which can introduce minor errors in generation probability calculations. Simulation helps to circumvent this.
  • Future Work: The findings suggest that future efforts should focus on understanding and curating training datasets to optimize their composition for better epitope-TCR prediction models. This includes carefully considering the balance between epitope-specific and negative data, and the inherent diversity within the epitope-specific repertoires.

Summary updated

Abstract

Abstract Numerous efforts have been made to decipher the epitope-T cell receptor (TCR) recognition code. Both simple machine learning techniques and deep learning strategies have been used to train models to predict the binding of epitopes by TCR sequences. A good training data set rests at the basis of every accurate prediction model, yet little attention has been given to the composition of these data sets. In this paper, we studied the natural distribution of TCR sequences within epitope-specific TCR repertoires, i.e. a set of TCRs binding the same epitope, and its impact on the predictability of TCR-epitope interactions. We found that the observed diversity of these repertoires can result from a smaller set of core binding motifs constrained by TCR generation. Moreover, a clear relationship was found between the sequence distribution of the training data and performance metrics, emphasizing the importance of the used ground-truth data when using machine learning models in this domain. Taken together, these findings inform data set composition to help push epitope-TCR prediction models to the next level.
BibTeX
@misc{gielis2024revealing,
  author = {Sofie Gielis and Maria Chernigovskaya and Milena Pavlović and Vincent Van Deuren and Romi Vandoren and Sebastiaan Valkiers and Kris Laukens and Victor Greiff and Pieter Meysman},
  title = {{Revealing the hidden sequence distribution of epitope-specific TCR repertoires and its influence on machine learning model performance}},
  year = {2024},
  doi = {10.1101/2024.10.21.619364},
  howpublished = {bioRxiv},
}
Figure