Abstract
Traditional metrics like the F1 score and Matthews Correlation Coefficient (MCC) for RNA secondary structure prediction are "all-or-nothing" measures that fail to capture topological similarities in "near-miss" predictions. This paper proposes the Weisfeiler-Lehman (WL) graph kernel as a biologically relevant alternative that accounts for structural shifts and sequence context. Our evaluation shows that the WL kernel provides a more nuanced assessment of model performance and can effectively guide RNA design algorithms toward better structural solutions.
Introduction
RNA molecules are vital regulators in biological systems, with their functions dictated by their complex 3D shapes. Predicting the secondary structure—the set of base pairs formed by hydrogen bonds—is a foundational task in computational biology. While deep learning has significantly advanced prediction capabilities, evaluation methodologies have remained stagnant, relying on binary classification metrics like F1 and MCC. These metrics treat any deviation from the ground truth as a total failure, ignoring the functional similarity of structures that are merely shifted by a few nucleotides.
The Weisfeiler-Lehman Graph Kernel for RNA
To address these limitations, we propose treating RNA secondary structures as graphs, where nucleotides are nodes and base pairs are edges. We employ the Weisfeiler-Lehman (WL) graph kernel to compare these structures. The WL algorithm uses iterative label propagation to create structural signatures for each node based on its local neighborhood. By counting shared signatures between two graphs, the WL kernel provides a similarity score sensitive to topology rather than just exact coordinate matching.
Sensitivity to Structural Shifts
A key advantage of the WL kernel is its robustness to "near-miss" predictions. In many biological contexts, a structure that is slightly shifted is functionally equivalent to the target. However, traditional metrics are highly sensitive to such shifts.
As illustrated in the structural shift experiments on the 5SrRNA of Drosophila melanogaster, a shift of just one or two positions causes F1 and MCC scores to plummet to zero or near-zero values. In contrast, the WL kernel exhibits a smooth decay, correctly identifying the high degree of topological similarity between the shifted and ground-truth structures.
Impact on Model Benchmarking
The choice of evaluation metric has profound implications for how we rank and develop RNA folding algorithms. We benchmarked 12 state-of-the-art models using traditional metrics alongside the WL kernel.
The results indicate significant leaderboard shifts. For instance, the RNA-FM model moves from 4th place under the F1 metric to 2nd place under the WL metric. This suggests that some models are better at capturing the overall structural topology than achieving exact bit-matching, a distinction that traditional metrics obscure.
Better Guidance for RNA Design
Beyond evaluation, the WL kernel can serve as a more effective objective function for generative tasks. When integrated into the reinforcement learning loop of the RNA design algorithm libLEARNA, the WL-guided agent (libLEARNA-WL) outperformed versions using standard distance measures on the Eterna100 benchmark. This indicates that the WL kernel provides a "smoother" and more informative optimization landscape for AI agents.
Conclusion and Future Work
The Weisfeiler-Lehman graph kernel offers a more biologically meaningful way to evaluate RNA secondary structure predictions. While the current implementation does not explicitly model thermodynamic stability or base-stacking, it represents a significant step toward more nuanced computational biology metrics. Future research will focus on developing differentiable GNN-based surrogate models to incorporate these structural insights directly into the training loss of deep learning architectures.
Summary updated