TL;DR
This paper reveals a crucial trade-off between the quantity and quality of sequence data when predicting protein fitness changes due to mutations. They show that simpler models, trained on carefully selected, high-quality (i.e., close to the wild-type) sequences, can outperform complex models trained on vast, diverse datasets. This work provides a theoretical framework and practical methods for optimizing data selection to maximize prediction accuracy.
What Problem Does This Paper Address?
Predicting how mutations affect protein function is vital for understanding evolution and diagnosing genetic diseases. High-throughput experiments can measure these effects, but the sheer number of possible mutations makes a complete "fitness landscape" map impossible for most proteins. Computational models, especially those based on evolutionary sequence data (like Multiple Sequence Alignments or MSAs), are used to predict these fitness effects. However, it's been unclear how the characteristics of the MSA data (how many sequences, how similar they are to the target protein) interact with model complexity to determine prediction accuracy. The authors aim to bridge this gap by theoretically analyzing and empirically validating the factors that govern prediction performance.
Methods
The core of the paper lies in a theoretical framework built upon the bias-variance trade-off from statistics. They propose using "mean Hamming distance" (D) as a measure of sequence data quality (how close sequences are to the wild-type) and "effective number of sequences" (B) as a measure of quantity. They analyze prediction error as a sum of squared bias (related to data quality and unmodeled complexity) and variance (related to data quantity and model complexity).
Key methodological contributions include:
- Theoretical Framework: Decomposing prediction error into bias and variance and relating them to data quality (D) and quantity (B).
- Bias Factor (Jo): Introducing a parameter that quantifies unmodeled epistasis (interactions between mutations) and links it to data quality.
- Sparse Potts Models: Using K-link Potts models, which allow for varying model complexity by controlling the number of pairwise interactions considered.
- "Focusing Procedure": A practical method for optimally selecting subsets of MSAs by iteratively removing sequences based on their Hamming distance to the wild-type.
- In Silico Lattice Protein Model: Utilizing a well-defined, solvable model to rigorously test their theoretical predictions.
Datasets
The authors evaluated their approach on:
- 7 Real Protein Families: Data was sourced from existing mutagenesis experiments and their corresponding MSAs, as described in Table 1 and Supplementary Figure 3. These include proteins like Beta-lactamase, DNA Binding domain, Ubiquitin, RNA Binding, PDZ, UBOX, and WW domains.
- In Silico Lattice Protein Models: A 27-site cubic lattice protein model was used for rigorous validation, where the "ground truth" fitness landscape is precisely known. This allowed for direct computation of bias and variance.
Key Insights and Ideas
- Quantity vs. Quality: More data isn't always better. High-quality data (sequences close to the wild-type) can be more valuable than a large quantity of diverse data, especially for simpler models.
- Simple Models Can Win: Surprisingly, simple models (like independent-site models) can outperform complex models when trained on optimally selected, high-quality subsets of data. This is because complex models can overfit noisy or distant data.
- Bias as a Signal of Epistasis: The "bias factor" (Jo) directly quantifies how much of the protein's fitness landscape complexity (specifically, epistatic interactions) is not captured by the chosen model. A higher Jo indicates more unmodeled epistasis.
- Focusing is Key: The "focusing procedure" of selecting sequences closer to the wild-type is a powerful way to improve prediction accuracy, especially for simpler models.
- Predictable Optimization: The optimal data subset for prediction can be reliably predicted, even without full knowledge of the fitness landscape, using heuristics based on signal-to-noise ratios.
Key Results
- Improved Prediction Accuracy: The focusing procedure, applied to independent-site models, led to significant improvements in prediction accuracy for 6 out of 7 protein families, with some exceeding the performance of more complex models trained on full MSAs. For the DNA-bind family, performance increased by 0.26.
- Strong Correlation: The theoretical framework, relating prediction performance (p) to the sum of squared bias (JoD) and variance (σ²), showed a strong correlation (rs > 0.95) across multiple protein families.
- Bias Factor Predictability: The bias factor Jo was shown to decrease with increasing model complexity (more learned couplings) and saturate, indicating that beyond a certain point, adding more parameters doesn't help capture unmodeled epistasis.
- Optimal Cutoff Prediction: The optimal cutoff distance for focusing could be predicted with good accuracy (mean relative error of 0.08), leading to systematic performance improvements.
Limitations and Future Work
- Model Choice: The study primarily focuses on Potts models. While they explore varying complexity within this class, extending the framework to other model types (e.g., deep learning) could be beneficial.
- Hamming Distance as Quality Metric: While effective, Hamming distance is a simplified measure of sequence quality. Future work could explore more sophisticated metrics, potentially incorporating phylogenetic information or functional relevance.
- Epistasis Complexity: The paper discusses higher-order epistasis but primarily models pairwise interactions. Investigating how to effectively model and predict effects involving three or more interacting sites remains an open challenge.
- Generalizability: While tested on 7 protein families and in silico models, further validation on a broader range of biological systems would strengthen the findings.
- Practical Application: The authors suggest that their framework can aid in selecting optimal datasets for predicting missense mutations relevant to genetic diseases, which often have limited and focused sequence data. They also propose its use in phylogenetic studies and other machine learning applications involving local regression.
Summary updated