Benchmarking tree and ancestral sequence inference for B cell receptor sequences

Kristian Davidsen, F. A. Matsen

TL;DR

This paper benchmarks phylogenetic inference tools for B cell receptors (BCRs), which evolve under unique biological processes like somatic hypermutation and affinity maturation. The authors developed a novel simulation framework to evaluate these tools, finding that current methods vary significantly in accuracy and that explicitly modeling BCR-specific mutation processes offers substantial performance gains.

What Problem Does This Paper Address?

B cell receptors (BCRs) are crucial for adaptive immunity, undergoing a complex evolutionary process called affinity maturation to improve antigen binding. This process involves mutation and selection, leading to clonal families of related BCR sequences. Phylogenetic methods are widely used to reconstruct the evolutionary history (phylogenetic trees) and infer ancestral sequences of these BCRs. However, BCR evolution has unique characteristics, such as context-dependent somatic hypermutation (SHM) and strong selective pressure, which differ from the assumptions of standard phylogenetic tools. It's unclear how well existing general-purpose phylogenetic methods perform on BCR data, and whether newer, BCR-specific tools offer significant improvements. This paper aims to benchmark these tools to understand their performance and identify areas for improvement.

Methods

The authors developed a novel forward-time simulation framework for BCR affinity maturation. This framework models:

  • Neutral Evolution: A baseline branching process where cells divide and mutate without selection.
  • Affinity Selection: A more realistic model where cell division and survival are driven by BCR affinity for an antigen, mimicking the germinal center environment. This model incorporates a kinetic model of BCR-antigen binding and a fitness landscape based on amino acid sequence distance to a "mature" sequence.
  • Somatic Hypermutation (SHM): Crucially, the simulation incorporates a context-sensitive SHM model (S5F motif model) that accounts for how the surrounding DNA sequence influences mutation rates, a key feature of BCR evolution.

They then used this simulator to generate synthetic BCR sequence data and benchmarked several phylogenetic inference tools, including general-purpose methods (PHYLIP's dnaml and dnapars, IQ-TREE) and BCR-specific tools (IgPhyML, GCtree). Performance was evaluated using metrics like the COAR (Correctness of Ancestral Reconstruction) score, MRCA (Most Recent Common Ancestor) distance, and RF (Robinson-Foulds) distance for tree topology. They also introduced an "isotype scoring" method using the irreversible nature of B cell isotype switching as an empirical validation of phylogenetic accuracy.

Datasets

The primary "dataset" used for evaluation was synthetically generated data from the authors' novel BCR sequence simulation framework. This framework was seeded with 288 naive BCR sequences inferred from a human single-cell dataset.

For validation with real data, they used BCR sequence data with isotype information from a previous study (Laustsen et al., 2016).

Key Insights and Ideas

  • BCR Evolution is Special: The unique aspects of BCR evolution (context-dependent SHM, strong selection) mean that general-purpose phylogenetic tools, which often assume independent site evolution and neutral drift, may not perform optimally.
  • Modeling SHM Matters: Explicitly modeling the context-sensitive nature of SHM significantly improves phylogenetic inference accuracy. Tools that incorporate this (like SAMM, which uses a motif-based ranking) show marked performance gains.
  • Selection Adds Complexity: Affinity maturation, driven by selection, makes the phylogenetic inference problem considerably harder than neutral evolution, leading to lower accuracy across all tested methods.
  • Implementation is Key: Even with similar underlying models, different software implementations can lead to substantial performance differences (e.g., dnaml vs. IQ-TREE).
  • Isotype Switching as a Constraint: The irreversible nature of isotype switching provides a valuable biological constraint that can be used to assess the plausibility of inferred phylogenies, though its high variance in current analyses limits its direct use as a primary metric.

Key Results

  • Tool Variability: The tested phylogenetic tools showed substantial variation in their ability to accurately infer BCR phylogenies and ancestral sequences.
  • SAMM Outperforms: The SAMM tool, which uses a motif-based likelihood ranking of parsimonious trees, performed best when using a motif model that matched the simulation's SHM model. This highlights the benefit of incorporating SHM motif bias.
  • GCtree's Potential: GCtree, which uses genotype abundance information to rank parsimonious trees, was the second-best performing tool, demonstrating the value of incorporating abundance data.
  • Selection vs. Neutral: Performance was consistently worse for affinity-selected simulations compared to neutral simulations across all metrics and tools.
  • Isotype Score Validation: While noisy, the isotype scoring method generally supported the findings that methods incorporating more biological realism (like SAMM and GCtree) produced phylogenies with fewer isotype switching violations than basic parsimony.

Limitations and Future Work

  • Naive Sequence Uncertainty: The study assumes the correct naive sequence is provided as an outgroup. The impact of uncertainty in naive sequence inference on downstream phylogenetic analysis was not directly tested but is identified as a crucial area for future work.
  • Indels Not Modeled: The simulations did not include insertion-deletion (indel) mutations, which are known to occur in BCR phylogenies. Current tools handle indels poorly, and their inclusion would be a valuable extension.
  • Alignment Step: The paper acknowledges that the alignment step itself can significantly impact ancestral sequence reconstruction accuracy, but this was not explicitly benchmarked.
  • Real Data Variance: The isotype scoring method, while promising, suffered from high variance in real data, limiting its direct use as a primary performance metric. Future work could explore better sampling strategies or more clonal families to improve its resolution.
  • Tool Availability: Some BCR-specific tools (e.g., ARPP) were not included due to availability issues.
  • Future Directions: The authors suggest developing methods that simultaneously perform phylogenetic reconstruction and naive sequence estimation, and incorporating indel mutations and alignment benchmarking into future studies.

Summary updated

Abstract

Abstract B cell receptor sequences evolve during affinity maturation according to a Darwinian process of mutation and selection. Phylogenetic tools are used extensively to reconstruct ancestral sequences and phylogenetic trees from affinity-matured sequences. In addition to using general-purpose phylogenetic methods, researchers have developed new tools to accommodate the special features of B cell sequence evolution. However, the performance of classical phylogenetic techniques in the presence of B cell-specific features is not well understood, nor how much the newer generation of B cell specific tools represent an improvement over classical methods. In this paper we benchmark the performance of classical phylogenetic and new B cell-specific tools when applied to B cell receptor sequences simulated from a forward-time model of B cell receptor affinity maturation towards a mature receptor. We show that the currently used tools vary substantially in terms of tree structure and ancestral sequence inference accuracy. Furthermore, we show that there are still large performance gains to be achieved by modeling the special mutation process of B cell receptors. These conclusions are further strengthened with real data using the rules of isotype switching to count possible violations within each inferred phylogeny.
BibTeX
@inproceedings{davidsen2018benchmarking,
  author = {Kristian Davidsen and F. A. Matsen},
  title = {{Benchmarking tree and ancestral sequence inference for B cell receptor sequences}},
  booktitle = {Frontiers in Immunology},
  year = {2018},
  doi = {10.3389/fimmu.2018.02451},
}
Figure