Rethinking Performance Measures of RNA Secondary Structure Problems

Frederic Runge, Jörg K. H. Franke, Daniel Fertmann, Frank HutterarXiv 2024 · DOI · arXiv

Abstract

Traditional metrics like the F1 score and Matthews Correlation Coefficient (MCC) for RNA secondary structure prediction are "all-or-nothing" measures that fail to capture topological similarities in "near-miss" predictions. This paper proposes the Weisfeiler-Lehman (WL) graph kernel as a biologically relevant alternative that accounts for structural shifts and sequence context. Our evaluation shows that the WL kernel provides a more nuanced assessment of model performance and can effectively guide RNA design algorithms toward better structural solutions.

Introduction

RNA molecules are vital regulators in biological systems, with their functions dictated by their complex 3D shapes. Predicting the secondary structure—the set of base pairs formed by hydrogen bonds—is a foundational task in computational biology. While deep learning has significantly advanced prediction capabilities, evaluation methodologies have remained stagnant, relying on binary classification metrics like F1 and MCC. These metrics treat any deviation from the ground truth as a total failure, ignoring the functional similarity of structures that are merely shifted by a few nucleotides.

The Weisfeiler-Lehman Graph Kernel for RNA

To address these limitations, we propose treating RNA secondary structures as graphs, where nucleotides are nodes and base pairs are edges. We employ the Weisfeiler-Lehman (WL) graph kernel to compare these structures. The WL algorithm uses iterative label propagation to create structural signatures for each node based on its local neighborhood. By counting shared signatures between two graphs, the WL kernel provides a similarity score sensitive to topology rather than just exact coordinate matching.

Sensitivity to Structural Shifts

A key advantage of the WL kernel is its robustness to "near-miss" predictions. In many biological contexts, a structure that is slightly shifted is functionally equivalent to the target. However, traditional metrics are highly sensitive to such shifts.

Comparison of F1, MCC, and WL scores for structural shifts in Drosophila melanogaster 5SrRNA.

As illustrated in the structural shift experiments on the 5SrRNA of Drosophila melanogaster, a shift of just one or two positions causes F1 and MCC scores to plummet to zero or near-zero values. In contrast, the WL kernel exhibits a smooth decay, correctly identifying the high degree of topological similarity between the shifted and ground-truth structures.

Impact on Model Benchmarking

The choice of evaluation metric has profound implications for how we rank and develop RNA folding algorithms. We benchmarked 12 state-of-the-art models using traditional metrics alongside the WL kernel.

Ranking of RNA folding algorithms across different metrics.

The results indicate significant leaderboard shifts. For instance, the RNA-FM model moves from 4th place under the F1 metric to 2nd place under the WL metric. This suggests that some models are better at capturing the overall structural topology than achieving exact bit-matching, a distinction that traditional metrics obscure.

Better Guidance for RNA Design

Beyond evaluation, the WL kernel can serve as a more effective objective function for generative tasks. When integrated into the reinforcement learning loop of the RNA design algorithm libLEARNA, the WL-guided agent (libLEARNA-WL) outperformed versions using standard distance measures on the Eterna100 benchmark. This indicates that the WL kernel provides a "smoother" and more informative optimization landscape for AI agents.

Conclusion and Future Work

The Weisfeiler-Lehman graph kernel offers a more biologically meaningful way to evaluate RNA secondary structure predictions. While the current implementation does not explicitly model thermodynamic stability or base-stacking, it represents a significant step toward more nuanced computational biology metrics. Future research will focus on developing differentiable GNN-based surrogate models to incorporate these structural insights directly into the training loss of deep learning architectures.

Summary updated

Abstract

Accurate RNA secondary structure prediction is vital for understanding cellular regulation and disease mechanisms. Deep learning (DL) methods have surpassed traditional algorithms by predicting complex features like pseudoknots and multi-interacting base pairs. However, traditional distance measures can hardly deal with such tertiary interactions and the currently used evaluation measures (F1 score, MCC) have limitations. We propose the Weisfeiler-Lehman graph kernel (WL) as an alternative metric. Embracing graph-based metrics like WL enables fair and accurate evaluation of RNA structure prediction algorithms. Further, WL provides informative guidance, as demonstrated in an RNA design experiment.
BibTeX
@misc{runge2024rethinking,
  author = {Frederic Runge and Jörg K. H. Franke and Daniel Fertmann and Frank Hutter},
  title = {{Rethinking Performance Measures of RNA Secondary Structure Problems}},
  year = {2024},
  eprint = {2401.05351},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2401.05351},
}
Figure