TL;DR
This paper introduces a more efficient and accurate method, "model-switch stepping-stone sampling," for comparing complex evolutionary models of DNA sequences. By directly estimating the Bayes factor between models, it avoids the pitfalls of estimating each model's marginal likelihood separately, leading to more reliable conclusions, especially for high-dimensional models.
What Problem Does This Paper Address?
Comparing evolutionary models is crucial for understanding biological processes, but it's computationally intensive, especially for complex, "high-dimensional" models that have many parameters. Traditional methods like the Harmonic Mean Estimator (HME) are biased towards these complex models, often overestimating their fit. Newer methods like Path Sampling (PS) and Stepping-Stone Sampling (SS) offer better accuracy but are computationally demanding. A key challenge is that estimating each model's marginal likelihood independently can accumulate errors, making the resulting Bayes factor unreliable, particularly when the true difference in model fit is small. This paper aims to improve the accuracy and efficiency of Bayes factor estimation for these complex models by developing a direct estimation method.
Methods
The core contribution is the development of a "model-switch" version of Stepping-Stone Sampling (SS). Unlike standard SS which estimates each model's marginal likelihood separately, this new approach constructs a direct path between two competing models in the space of unnormalized densities. This "model-switch" strategy bypasses the need for independent marginal likelihood calculations, thereby reducing accumulated errors. The authors also explore different "sigmoid shape" parameters to control the path taken between models, aiming to find an optimal balance between computational cost and accuracy. They compare this new method against the original "model-switch" Path Sampling (PS) and standard PS and SS approaches.
Datasets
The study evaluates their methods on three distinct datasets:
- Laurasiatheria dataset: A large mammalian dataset (97,682 nucleotides) from a conserved non-coding region, chosen for its potential to show significant model fit improvements due to CpG-methylation-deamination processes.
- Pseudogenes dataset: A smaller mammalian dataset (6,166 nucleotides) from primate $\psi$-globin pseudogene sequences, used to assess performance with shorter sequences and fewer data points.
- Nuclear SSU rRNA dataset: A plant dataset (1,619 nucleotides) from small subunit ribosomal RNA genes, selected because context-dependent evolutionary processes are not expected to be as prominent, serving as a test for how well the models penalize excessive parameters.
Key Insights and Ideas
- Direct Bayes Factor Estimation is Superior: Directly estimating the Bayes factor by creating a path between models (model-switch PS/SS) is statistically preferable to estimating each model's marginal likelihood independently. This reduces error accumulation, especially when the true difference in model fit is subtle.
- Stepping-Stone Sampling Outperforms Path Sampling: The proposed "model-switch" Stepping-Stone Sampling (SS) method generally shows lower variance and better accuracy than the "model-switch" Path Sampling (PS) approach, especially when computational resources are limited.
- Harmonic Mean Estimators are Biased: The study reinforces previous findings that Harmonic Mean Estimators (HME and sHME) are biased towards higher-dimensional models, leading to overestimations of model fit and potentially erroneous conclusions, particularly for plant DNA sequences where context-dependency is less pronounced.
- Computational Cost is Key: Achieving accurate Bayes factor estimates for high-dimensional models requires substantial computational effort. While shorter runs can give a rough idea of model fit, truly reliable estimates demand extensive computation (e.g., thousands of path steps and many MCMC iterations per step).
- Sigmoid Shape Matters: The choice of the "sigmoid shape" parameter, which defines the path between models, significantly impacts the accuracy and bidirectional error of the estimates. Values between 9.0 and 12.0 generally provided the best balance for the tested datasets.
Key Results
- For the Laurasiatheria dataset, the new model-switch SS method provided more stable and reliable log Bayes factor estimates compared to PS, especially with increased computational settings (K=16,000 path steps). The HME/sHME estimates were drastically different from PS/SS estimates, showing a log Bayes factor of ~3522.93 for the context-dependent model over GTR, while PS/SS suggested a difference of ~2921.
- On the Pseudogenes dataset, the context-dependent model also showed a better fit than the GTR model, with log Bayes factors around 175.12 using SS.
- For the Nuclear SSU rRNA plant dataset, the HME/sHME incorrectly favored the more complex context-dependent model, yielding a log Bayes factor of 54.64. In contrast, the SS method correctly indicated a poorer fit for the context-dependent model, with log Bayes factors around -94.23, demonstrating its ability to penalize models with excessive parameters when they don't improve fit.
- The study found that increasing the number of path steps (K) was more critical than increasing the number of MCMC iterations per step (Q) for improving accuracy, particularly for the SS method.
Limitations and Future Work
- Computational Demands: The primary limitation is the significant computational cost associated with achieving accurate estimates, especially for very high-dimensional models. The paper highlights that even extensive settings can require days of computation on multi-core processors.
- "Model-Switch" is Computationally Intensive: While more accurate, the "model-switch" approach still requires substantial computation.
- Future Work: The authors mention that extending "generalized stepping-stone sampling" to directly construct a path between two competing models is an area of ongoing research. This could potentially offer further improvements in efficiency and accuracy. They also suggest that the choice of sigmoid shape parameter might be further optimized or automated.
Summary updated