Abstract
Phylogenetic mixtures model the inhomogeneous molecular evolution commonly observed in data. The performance of phylogenetic reconstruction methods where the underlying data is generated by a mixture model has stimulated considerable recent debate. Much of the controversy stems from simulations of mixture model data on a given tree topology for which reconstruction algorithms output a tree of a different topology; these findings were held up to show the shortcomings of particular tree reconstruction methods. In so doing, the underlying assumption was that mixture model data on one topology can be distinguished from data evolved on an unmixed tree of another topology given enough data and the "correct" method. Here we show that this assumption can be false. For biologists our results imply that, for example, the combined data from two genes whose phylogenetic trees differ only in terms of branch lengths can perfectly fit a tree of a different topology.
Introduction
In the field of phylogenetics, the primary goal is to reconstruct the "true" tree topology that describes the evolutionary relationships between species. However, molecular evolution is rarely a uniform process. Different genes, or even different regions within a single gene, often evolve at varying rates. This heterogeneity is typically modeled using "phylogenetic mixtures."
A significant challenge arises when reconstruction methods, such as Maximum Likelihood, yield an incorrect tree topology from mixture data. Historically, researchers attributed these failures to algorithmic limitations, assuming that sufficient data and refined methods would eventually reveal the true tree. This paper by Matsen and Steel (2007) addresses the more fundamental issue of identifiability: whether the data itself contains enough information to distinguish between competing tree structures. The authors demonstrate that in certain scenarios, the answer is a definitive "no."
Methodology
The study focuses on the two-state symmetric model (also known as the Cavender-Farris-Neyman model), a simplified representation of nucleotide or protein changes. To analyze the behavior of mixtures, the authors utilize two core mathematical frameworks:
- The Hadamard Transform: This mapping converts branch lengths (evolutionary change) into expected frequencies of "site patterns" (the distribution of character states across species).
- Phylogenetic Invariants: These are polynomial equations that must equal zero for a specific tree topology. If data satisfies the invariants for one topology but not another, the former is considered the correct fit.
By exploring the parameter space of these equations, the authors identified regions where a weighted average (mixture) of two sets of branch lengths on one tree topology satisfies the invariants of a completely different topology.
Key Insights: Mixed Branch Repulsion and Perfect Mimicry
The authors introduce the concept of Mixed Branch Repulsion. This phenomenon occurs when specific branch length patterns—such as neighboring branches alternating between long and short states—cause those branches to "repel" each other during reconstruction, favoring an incorrect topology.
Crucially, the paper proves the existence of Perfect Mimicry. This is not merely a case of data being "noisy" or "close" to another tree; rather, the site pattern frequencies generated by a mixture on one tree can be mathematically identical to those produced by an unmixed process on a different tree. This finding implies that the assumption of "generic identifiability"—the idea that the true tree is almost always findable—fails in specific, characterizable regions of the parameter space.
Key Results
The authors demonstrated that these "pathological" cases are not isolated anomalies but exist within a full-dimensional region of the parameter space. This suggests that while such conditions may be rare, they are mathematically robust and potentially encounterable in real-world biological data.
Specifically, for a 4-species (quartet) tree, the authors derived a set of three inequalities that define the "Zone of Repulsion" where a mixture will mimic a different topology.
Furthermore, the study provides a geometric perspective, showing that the "convex hull" (the set of all possible mixtures) of data varieties for one topology can intersect the variety of a different topology. This intersection represents the point of perfect mimicry.
Conclusion and Future Work
While the proof relies on the two-state symmetric model, the authors suggest these results likely extend to more complex models like GTR or Jukes-Cantor. The study was also limited to 4-taxon trees; subsequent research is needed to investigate how mixed branch repulsion scales with larger numbers of species.
For biologists, these findings serve as a cautionary note: when sequence data is ambiguous due to potential mixtures, relying solely on standard reconstruction methods may be insufficient. The authors suggest incorporating alternative data types—such as gene order or rare genomic markers—to resolve the true evolutionary history when sequence-based identifiability is compromised.
Summary updated