Multi-modal generative modeling for joint analysis of single-cell T cell receptor and gene expression data

Felix Drost, Yang An, Irene Bonafonte-Pardàs, Lisa M. Dratva, Rik G. H. Lindeboom, Muzlifah Haniffa, Sarah A. Teichmann, Fabian Theis, Mohammad Lotfollahi, Benjamin SchubertNature Communications 2024 · OpenReview

Abstract

Recent advances in single-cell immune profiling have enabled the simultaneous measurement of transcriptome and T cell receptor (TCR) sequences, offering great potential for studying immune responses at the cellular level. However, integrating these diverse modalities across datasets is challenging due to their unique data characteristics and technical variations. Here, to address this, we develop the multimodal generative model mvTCR to fuse modality-specific information across transcriptome and TCR into a shared representation. Our analysis demonstrates the added value of multimodal over unimodal approaches to capture antigen specificity. Notably, we use mvTCR to distinguish T cell subpopulations binding to SARS-CoV-2 antigens from bystander cells. Furthermore, when combined with reference mapping approaches, mvTCR can map newly generated datasets to extensive T cell references, facilitating knowledge transfer. In summary, we envision mvTCR to enable a scalable analysis of multimodal immune profiling data and advance our understanding of immune responses.

Introduction

T cells are the "special ops" of the adaptive immune system, with their identity defined by two key features: their phenotype (functional state, captured by gene expression) and their specificity (what they recognize, determined by the T cell receptor or TCR). While modern single-cell sequencing can measure both simultaneously, integrating these modalities is computationally difficult. Gene expression data is continuous and high-dimensional, whereas TCR data consists of discrete amino acid sequences. Existing tools often struggle to scale to "atlas-level" datasets comprising hundreds of thousands of cells.

To bridge this gap, researchers have developed mvTCR, a multimodal deep learning model that fuses TCR sequences and gene expression data into a single, shared latent representation. By translating these distinct biological languages into a unified mathematical space, mvTCR outperforms existing tools in predicting antigen specificity and scales efficiently to massive datasets.

Methods: The mvTCR Architecture

The mvTCR model is built on a Variational Autoencoder (VAE) framework designed to handle the disparate nature of transcriptomic and proteomic sequence data.

The mvTCR architecture uses specialized encoders for RNA and TCR data, fused via a Mixture-of-Experts module.

As shown in Figure 1, the model employs two specialized input channels:

  1. RNA Encoder: A Multi-Layer Perceptron (MLP) that compresses high-dimensional gene expression data.
  2. TCR Encoder: A Transformer network that captures structural patterns within the CDR3$\alpha$ and CDR3$\beta$ amino acid sequences.

A Mixture-of-Experts (MoE) fusion module serves as the core integration component, learning to dynamically weight the contribution of each modality for every cell. This process creates a shared latent space where cells with similar functions and receptors cluster together. The model also generates a "TCR-contribution" score, providing interpretability by indicating whether a cell's identity is driven more by its receptor or its transcriptional state.

Key Results and Insights

The authors validated mvTCR across nine datasets, including high-quality antigen-binding benchmarks, large-scale SARS-CoV-2 studies, and a massive Tumor Infiltrating Lymphocyte (TIL) atlas of over 700,000 cells.

Multimodal Synergy and Specificity

A primary finding is that the whole is greater than the sum of its parts. mvTCR significantly outperformed unimodal approaches in predicting antigen specificity, achieving an average F1-score of 0.82, compared to 0.76-0.77 for models using only TCR or RNA data.

Comparison of latent spaces generated by TCR-only, RNA-only, and multimodal mvTCR approaches demonstrates superior clustering by specificity.

As illustrated in Figure 5, the joint mvTCR representation (right) provides much clearer separation of cells by antigen specificity compared to TCR-only or RNA-only UMAPs. This synergy allows for the identification of "bystander" cells—T cells that appear activated but do not actually bind to target antigens (e.g., SARS-CoV-2)—a distinction that is often invisible in transcriptomic data alone.

Scalability and Reference Mapping

mvTCR demonstrates remarkable computational efficiency. In benchmarks, it was up to 135x faster than CoNGA and 2,350x faster than tessa when processing 100,000 cells. Furthermore, by integrating with scArches, mvTCR enables "reference mapping," allowing researchers to project new, smaller datasets onto massive existing T cell atlases without retraining the entire model from scratch.

Discussion and Future Directions

While mvTCR represents a significant advance in multimodal immune profiling, several avenues for future work remain. Currently, adjusting the relative importance of TCR vs. RNA requires retraining; future iterations may allow for more dynamic hyperparameter tuning. Additionally, while the current model focuses on the CDR3 regions, expanding the scope to full VDJ segments could provide even greater resolution. Finally, the architecture is theoretically adaptable to B cell receptors (BCRs), which could revolutionize vaccine research and antibody discovery.

Reference: Drost, F., An, Y., et al. (2024). Multi-modal generative modeling for joint analysis of single-cell T cell receptor and gene expression data. Nature Communications. https://doi.org/10.1038/s41467-024-49806-9

Summary updated

Abstract

Recent advances in single-cell immune profiling have enabled the simultaneous measurement of transcriptome and T cell receptor (TCR) sequences, offering great potential for studying immune responses at the cellular level. However, integrating these diverse modalities across datasets is challenging due to their unique data characteristics and technical variations. Here, to address this, we develop the multimodal generative model mvTCR to fuse modality-specific information across transcriptome and TCR into a shared representation. Our analysis demonstrates the added value of multimodal over unimodal approaches to capture antigen specificity. Notably, we use mvTCR to distinguish T cell subpopulations binding to SARS-CoV-2 antigens from bystander cells. Furthermore, when combined with reference mapping approaches, mvTCR can map newly generated datasets to extensive T cell references, facilitating knowledge transfer. In summary, we envision mvTCR to enable a scalable analysis of multimodal immune profiling data and advance our understanding of immune responses. Although single-cell RNA sequencing analysis now allows simultaneous examination of transcriptome and T cell receptor repertoire sequences, integrating these two modalities remains a challenge. Here, the authors develop mvTCR, a generative deep learning model that integrates transcriptome and T cell receptor data into a joint representation capturing cell functions and phenotypes.
BibTeX
@inproceedings{drost2024multi,
  author = {Felix Drost and Yang An and Irene Bonafonte-Pardàs and Lisa M. Dratva and Rik G. H. Lindeboom and Muzlifah Haniffa and Sarah A. Teichmann and Fabian Theis and Mohammad Lotfollahi and Benjamin Schubert},
  title = {{Multi-modal generative modeling for joint analysis of single-cell T cell receptor and gene expression data}},
  booktitle = {Nature Communications},
  year = {2024},
  url = {https://openreview.net/forum?id=rUroGK4N8D},
}
Figure