Uneven gains in protein-RNA affinity and mutation-effect prediction

Shangkun Sima, Tongfei Shen, Tianchi LuarXiv 2026 · arXiv

TL;DR

Improving a model's ability to predict overall protein–RNA binding affinity does not guarantee it will be better at predicting the specific effects of mutations. This study demonstrates that gains in "aggregate" accuracy often stem from better estimates of average landscape effects rather than improved "local" accuracy—the ability to distinguish and rank specific mutations within a single protein system.

What Problem Does This Paper Address?

In the world of protein engineering and molecular biology, we often want to know how a single amino-acid change will affect how a protein binds to RNA. We use computational models to predict binding affinity ($pK_d$), assuming that if a model gets better at predicting absolute affinity across many different complexes, it will naturally get better at predicting the change in affinity ($\Delta pK_d$) caused by a mutation.

However, this isn't always true. Errors in predicting the "wild-type" (WT) and "mutant" endpoints can cancel each other out or compound in complex ways. A model might correctly predict that mutations generally weaken binding (improving its "average" score) while remaining completely unable to tell you which of two specific mutations is more damaging. The authors set out to decompose these errors and find out where modern models are actually succeeding—and where they are just "getting lucky" on the averages.

Methods

The researchers evaluated eleven different method families, ranging from sequence-only models to complex multimodal structural networks. Their primary vehicle for analysis was CoPRA (Complex-pretrained Protein-RNA Affinity), which they used in a controlled configuration to isolate specific effects.

The core of their approach is a rigorous error decomposition:

  1. Shared vs. Difference Error: They separated errors that are common to both the WT and mutant (which cancel out when calculating $\Delta pK_d$) from errors unique to the mutation effect itself.
  2. Landscape-Mean vs. Centered Local Error: They distinguished between a model's ability to predict the average effect of all mutations in a "landscape" (a specific protein-RNA system) versus its ability to predict the variation between those mutations.

They also used "feature tracing" in the PRA-Pred model and input controls in M2-PRNet to identify "computational restrictions"—cases where the model's architecture or data processing steps accidentally suppressed the signal of a mutation before it could even be evaluated.

Datasets

The study relied on ProNAB, a database of protein-nucleic acid binding affinities. They meticulously filtered this data to create "matched" populations:

  • Absolute Cohort: 752 units for testing general affinity.
  • Ordinary Pairs: 338 pairs linking a mutation to a WT anchor under identical experimental conditions.
  • Local Pairs: 220 pairs across 20 protein families, requiring at least five distinct mutations per system to evaluate local ranking and spread.
  • External Test Set: An independent set of 24 quantitative pairs from three source-matched families (SF3A1, SEPSECS, and Npl3) to validate how well these "gains" transfer to new data.

Key Insights and Ideas

  • The "Average" Trap: A model can show a lower Mean Absolute Error (MAE) simply by being better at predicting that "mutations usually decrease affinity." This improves the "landscape mean" but doesn't help a researcher who needs to prioritize which specific residue to mutate.
  • Response Amplitude (The "Timid" Model): Many models are "timid"—they predict very small changes in affinity even when the experimental change is large. While CoPRA generated stronger responses (higher SD ratio) than simpler additive models, this didn't always translate to better ranking.
  • Data Processing Can Kill Signal: In models like M2-PRNet, the way inputs are aligned or masked can actually "zero out" the difference between a WT and a mutant before the neural network even sees the data.
  • Uneven Transfer: Gains in absolute affinity prediction transfer "unevenly." As seen in Figure 1, a model might improve its pooled endpoint error across different systems without improving the "spread" or "order" of mutations within a single system.
Figure 1. Affinity and mutation-effect evaluation. a, RNase E–ssRNA (2C4R) and the 2.80 Å Phe57–G10 contact. b, Cross-system endpoints and within-system substitutions sharing a WT and assay. c, Same-member model comparisons, PRA-Pred regression terms and M2-PRNet branches.

Key Results

  • Absolute Gains: Controlled CoPRA reduced absolute-affinity MAE from 1.2898 to 1.1431 $pK_d$.
  • Mutation Effects: Across 220 mutation pairs, CoPRA improved mean-effect accuracy but showed no decrease in mean centered local error ($cMAE$). In fact, the $cMAE$ slightly worsened in the main cohort (from 0.3749 to 0.3991 $pK_d$), though it improved in the independent external test.
  • Ordering and Amplitude: CoPRA significantly increased the "boldness" of predictions. The ratio of predicted-to-observed standard deviation (SD ratio) rose from a nearly flat 0.0709 to 0.6422. However, the improvement in ranking (Spearman correlation) was modest and statistically uncertain.
  • Error Localization: The error decomposition (Figure 2/4) showed that most of the "gains" were localized in the landscape means, confirming that the model was learning general trends rather than fine-grained local variations.
Figure 2. Prediction errors, response amplitude and ordering. a, Errors and paired reductions on 752 absolute units, 338 ordinary pairs and 220 local pairs. Diamonds are family-macro means and bars are 95% family-bootstrap intervals. b, Local-scope reductions across 21 landscapes and 20 families, scaled by column. c, SD ratio (log radius), position Spearman (angle) and 50%/95% bootstrap mean-density contours. See Table 2.

Limitations and Future Work

The authors highlight a significant "heterogeneity" in results—the model might perform brilliantly on one protein family (like Hfq) but poorly on another. This suggests that current models are sensitive to the specific structural and chemical context of the binding site.

Future work should move away from reporting single aggregate metrics like MAE. Instead, the authors argue for a joint evaluation of average effects, mutation ordering, and quantitative local variation. Until a model can demonstrate it can rank mutations reliably within a single system, its utility for "prioritizing mutations" in the lab remains limited.

Summary updated

Abstract

Accurate matched affinities determine mutation effects, but gains in aggregate affinity prediction may transfer unevenly to average effects, ordering and local accuracy. We evaluated eleven method families using method-specific matched populations. Controlled CoPRA reduced absolute-affinity mean absolute error from 1.2898 to 1.1431 pKd. Across 220 mutation pairs from 20 families, it generated stronger responses and improved mean-effect accuracy, with uncertain ranking gains and no decrease in mean centered local error. In an independent external test (24 quantitative pairs from three source-matched families), absolute and ordinary mutation-effect errors fell in all three families. Mean centered local error fell from 0.4198 to 0.3761 pKd, with improvements in two families and higher error in one. Error decomposition localized gains in landscape means. PRA-Pred feature tracing, fixed-weight M2-PRNet input controls and matched supervision interventions identified specific restrictions on mutation response. These results establish the need to evaluate average effects, ordering and quantitative local variation jointly.
BibTeX
@misc{sima2026uneven,
  author = {Shangkun Sima and Tongfei Shen and Tianchi Lu},
  title = {{Uneven gains in protein-RNA affinity and mutation-effect prediction}},
  year = {2026},
  eprint = {2611.24739},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.24739},
}

Loading related content...

Figure