Abstract
Incomplete lineage sorting (ILS) is a common source of gene tree incongruence in multilocus analyses. A large number of methods have been developed to infer species trees in the presence of ILS. Here we provide a mathematical analysis of several coalescent-based methods. Our analysis is performed on a three-taxon species tree and assumes that the gene trees are correctly reconstructed along with their branch lengths.
Introduction
In evolutionary biology, the history of a single gene (a gene tree) often diverges from the history of the species from which it is derived. This phenomenon, known as Incomplete Lineage Sorting (ILS), occurs when ancestral gene lineages fail to coalesce before a speciation event. While numerous "multilocus" methods have been developed to infer species trees despite this noise, most comparisons between these methods have historically relied on computer simulations.
This paper provides a rigorous mathematical comparison of how different phylogenetic methods handle ILS. By applying large-deviations theory to a three-taxon model, the study quantifies the efficiency of various methods, determining how quickly the probability of error decreases as the number of sampled genes increases.
The Analytical Framework: Large-Deviations Theory
The study focuses on a three-taxon species tree (taxa A, B, and C) under the multispecies coalescent model. To compare methods, the author calculates the exponential decay rate ($\alpha$) of the failure probability. If $L$ represents the number of genes (loci), the probability of a method failing to identify the correct species tree is approximately $e^{-L\alpha}$.
A higher $\alpha$ indicates a more "efficient" method, meaning it requires fewer genes to reach a correct conclusion. The analysis categorizes seven popular methods into three distinct mathematical groups based on their behavior in the three-taxon case:
- ML, GLASS, and MT: Maximum Likelihood and methods based on minimum coalescence times.
- $R^*$, STAR, and MDC: Consensus methods, rank-based methods, and "minimizing deep coalescences."
- STEAC and SC: Methods based on average coalescence times.
Key Insights: The Power of the Minimum
One of the most significant findings is the superior performance of Maximum Likelihood (ML) and methods like GLASS/MT. In the three-taxon setting, these "minimum-seeking" methods are incredibly powerful because they succeed if even a single gene tree correctly captures the internal branch coalescence. This makes them exceptionally efficient when the internal branch is short and ILS is high.
In contrast, methods like STAR and STEAC rely on averages across all genes. While averaging is often seen as a robust statistical approach, this study demonstrates that these averages are heavily influenced by the "noise" of ILS. Consequently, these methods converge on the correct tree much more slowly than ML-based approaches. Furthermore, the paper proves that several seemingly different algorithms (like STAR and MDC) actually behave identically when limited to three taxa, simplifying the comparative landscape.
Comparative Results and Efficiency
The analysis establishes Maximum Likelihood as the gold standard, with a decay rate of $\alpha_{ML}(t) = t$. It remains the most efficient method across all branch lengths.
The difference in efficiency is most pronounced when ILS is high (where the internal branch length $t$ approaches zero). In these "hard" cases, the failure probability of ML decays linearly with $t$, whereas other methods decay quadratically ($t^2$). For example, with 500 genes and a short branch ($t=0.1$):
- ML/GLASS/MT failure probability is virtually zero ($\approx 1.9 \times 10^{-22}$).
- $R^*$/STAR/MDC failure probability is approximately 3.8%.
- STEAC/SC failure probability is approximately 16%.
The study also identifies a "crossover" effect: while consensus methods ($R^*$) outperform average-time methods (STEAC) when ILS is very high, STEAC becomes superior as the internal branch length increases, eventually tracking the optimal ML rate.
Limitations and Future Directions
The primary limitation of this analysis is the assumption of "perfect" gene trees—the idea that gene trees and their branch lengths are reconstructed without error. In practice, DNA sequence data is noisy. While GLASS and MT are mathematically optimal in this model, they rely on minimum values, which may make them highly sensitive to sequence errors in real-world applications. ML is expected to be more robust to such errors.
Furthermore, the analysis is currently limited to the three-taxon case. Extending this large-deviations framework to larger trees (four or more taxa) remains a significant mathematical challenge. Future research is needed to incorporate gene tree estimation error into this framework, bridging the gap between theoretical optimality and practical bioinformatics.
Summary updated