Benchmarking computational methods for B-cell receptor reconstruction from single-cell RNA-seq data

Tommaso Andreani, Linda M. Slot, Samuel Gabillard, Carsten Struebing, C. Reimertz, Yaligara Veeranagouda, Aleida M. Bakker, Reza Olfati-Saber, René E. M. Toes, Hans Ulrich Scherer, Franck Augé, Deimantė ŠimaitėBiorxiv 2022 · DOI

TL;DR

This paper benchmarks six computational tools for reconstructing full-length B-cell receptors (BCRs) from single-cell RNA sequencing (scRNA-seq) data. It provides crucial guidance for researchers by evaluating tool performance across various sequencing depths, read lengths, and levels of somatic hypermutation, identifying BALDR, BASIC, and BRACER as top performers.

What Problem Does This Paper Address?

The rapid advancement of single-cell RNA sequencing (scRNA-seq) has opened new avenues for studying B-cell receptors (BCRs), which are critical for understanding immune responses, antibody-based drug development, and diseases like cancer and autoimmune disorders. However, reconstructing full-length BCRs from scRNA-seq data is challenging due to factors like sequencing depth, read length, and the high variability introduced by somatic hypermutations (SHMs). Currently, there's a lack of comprehensive assessments to guide researchers in selecting the most appropriate computational tool for their specific data and research question. This paper aims to fill that gap by systematically evaluating and comparing six popular BCR reconstruction methods.

Methods

The authors developed a comprehensive benchmarking framework that involved:

  • Simulated and Experimental Datasets: They utilized three publicly available experimental datasets (Canzar, Upadhyay, and a newly generated "Leiden" dataset) and one simulated dataset. These datasets were manipulated to simulate varying read coverages and lengths.
  • Performance Metrics: Sensitivity was used for experimental data, and accuracy for simulated data.
  • Validation: Reconstructed BCRs were experimentally validated by producing monoclonal antibodies and testing their antigen specificity.
  • Execution Time: The runtime of each tool was measured to assess scalability.
  • V-(D)-J Gene Overlap: This metric was used to assess how consistently different tools identified the same gene segments, independent of ground truth.
  • Final Scoring: A cumulative average score was computed to rank the methods.

The six tools evaluated were MIXCR, BASIC, BALDR, VDJpuzzle, BRACER, and TRUST4.

Datasets

The study used the following datasets:

  • Leiden Dataset: A newly generated dataset from tetanus toxoid-specific and unknown specificity B-cells, sequenced using SMART-seq2.
  • Canzar Dataset: Publicly available data from plasmablasts (SPEC-seq).
  • Upadhyay Dataset: Publicly available data from plasmablasts (SMART-seq2).
  • Simulated Dataset: A synthetic dataset generated to specifically test the impact of varying levels of somatic hypermutations (SHMs) on BCR reconstruction.

These datasets were downsampled and trimmed to create numerous libraries with different read coverages (50,000 to 1,250,000 reads) and read lengths (25 bp to 100 bp) to thoroughly assess tool performance.

Key Insights and Ideas

  • No One-Size-Fits-All: The performance of BCR reconstruction tools is highly dependent on data characteristics like read length, coverage, and the presence of SHMs.
  • De Novo Assembly Shines for SHMs: Tools employing "de novo" assembly (like BRACER and BALDR) are generally more accurate in reconstructing BCRs with high levels of somatic hypermutations, which are common in memory B-cells and relevant for many disease studies.
  • Short Reads are a Challenge: Very short reads (e.g., 25 bp) significantly impact tool performance, with only BASIC demonstrating acceptable results in such scenarios.
  • Experimental Validation is Crucial: In silico reconstruction is a powerful first step, but experimental validation of antibody specificity is essential to confirm the biological relevance of the reconstructed BCRs.
  • Speed vs. Accuracy Trade-offs: While some tools are faster (TRUST4, MiXCR, BASIC), others offer higher accuracy, particularly for complex datasets (BRACER, BALDR).

Key Results

  • Best Overall Performers: BALDR, BASIC, and BRACER consistently showed the best overall performance across various datasets and conditions.
  • SHM Tolerance: BRACER and BALDR were the most accurate in reconstructing BCRs with varying degrees of SHMs.
  • Short Read Performance: BASIC was the only method that provided acceptable results on very short read libraries (25 bp).
  • Speed: TRUST4, MiXCR, and BASIC were the fastest tools.
  • Experimental Validation: Three reconstructed BCRs were successfully validated, producing monoclonal antibodies that specifically bound to tetanus toxoid, confirming the biological relevance of the computational reconstructions.
  • Read Length Impact: BRACER struggled with short reads (50 bp), while BALDR and TRUST4 maintained good performance across different read lengths.

Limitations and Future Work

  • Plate-Based Focus: This benchmark primarily focused on plate-based scRNA-seq techniques (SMART-seq2, SPEC-seq). The authors acknowledge that other popular platforms like 10x Genomics were not included due to challenges in obtaining ground truth data and limited tool availability for that platform.
  • Simulated Data Limitations: While simulated data is valuable for controlled experiments (like SHM impact), it cannot fully replicate the complexities of in vivo biological scenarios.
  • Future Directions: The authors suggest that future work should include benchmarking on other sequencing technologies (e.g., 10x, Oxford Nanopore, PacBio) and further developing methods to handle the challenges presented by these platforms. They also propose guidelines to help users select the best method based on their specific data characteristics and research questions.

Summary updated

Abstract

ABSTRACT Multiple methods have recently been developed to reconstruct full-length B-cell receptors (BCRs) from single-cell RNA-seq (scRNA-seq) data. This need emerged from the expansion of scRNA-seq techniques, the increasing interest in antibody-based drug development and the importance of BCR repertoire changes in cancer and autoimmune disease progression. However, a comprehensive assessment of performance-influencing factors like the sequencing depth, read length or the number of somatic hypermutations (SHMs) as well as guidance regarding the choice of methodology are still lacking. In this work, we evaluated the ability of six available methods to reconstruct full-length BCRs using one simulated and three experimental SMART-seq datasets. In addition, we validated that the BCRs assembled in silico recognize their intended targets when expressed as monoclonal antibodies. We observed that methods like BALDR, BASIC and BRACER showed the best overall performance across the tested datasets and conditions whereas only BASIC demonstrated acceptable results on very short read libraries. Furthermore, the de novo assembly-based methods BRACER and BALDR were the most accurate in reconstructing BCRs harboring different degrees of SHMs in the variable domain, while TRUST4, MiXCR and BASIC were the fastest. Finally, we propose guidelines to select the best method based on the given data characteristics.
BibTeX
@misc{andreani2022benchmarking,
  author = {Tommaso Andreani and Linda M. Slot and Samuel Gabillard and Carsten Struebing and C. Reimertz and Yaligara Veeranagouda and Aleida M. Bakker and Reza Olfati-Saber and René E. M. Toes and Hans Ulrich Scherer and Franck Augé and Deimantė Šimaitė},
  title = {{Benchmarking computational methods for B-cell receptor reconstruction from single-cell RNA-seq data}},
  year = {2022},
  doi = {10.1101/2022.03.24.485600},
  howpublished = {Biorxiv},
}

Loading related content...

Figure