Deep networks that predict how an RNA molecule folds onto itself now score near-perfectly on standard benchmarks. That includes pseudoknots, the tangled folds where pairs of bases cross each other. Yet these networks collapse on RNA families they have never seen. This note proposes treating that failure the way natural-language processing treats prompt brittleness. A trained folding model may be keying on family-specific surface features of the sequence, much as a language model keys on the exact wording of an instruction. The remedy in both cases is to train on "paraphrases": inputs that differ on the surface but mean the same thing. For RNA, a paraphrase is a sequence that folds into the same structure. Compensatory mutations produce such sequences: swapping a G–C pair for C–G or A–U changes the letters but leaves the pairing pattern intact.
The specific bet comes from a recent language-model result. That work found that semantically equivalent training prompts are not interchangeable. They produce very different updates, and the useful ones can be identified cheaply as those the current model already finds easy, meaning they have low loss before the update. We ask whether the same holds for RNA paraphrases. Concretely: does choosing, at each step, the structure-preserving variant with the lowest current loss beat random augmentation and hard-example mining on unseen-family pseudoknot prediction? And does it make the weight on the augmentation loss less fragile to tune? This is a sharp test, because the two obvious alternatives predict opposite things. Hard-example mining says pick the highest-loss variant. The prompt-selection result says pick the lowest.
Background and the gap
RNA secondary structure is the set of base pairs a single strand forms with itself. When pairs never cross, dynamic programming finds the lowest-energy fold in cubic time. This is why classic tools such as the ViennaRNA Package ignore pseudoknots. With arbitrary crossings, the problem becomes NP-hard (Predicting RNA Secondary Structures with Arbitrary Pseudoknots by Maximizing the Number of Stacking Pairs). Modern learned predictors avoid grammars altogether. They predict the full L×L contact matrix, a grid saying which positions pair, and then enforce validity in different ways:
- E2Efold unrolls a constrained optimizer into the network.
- RFold uses row and column softmaxes, so each base has at most one partner by construction.
- NTFold refines the contact map with a U-Net.
- DEPfold treats pseudoknots as crossing arcs in a dependency parse.
- PriFold adds pairing-frequency priors and covariation-based data augmentation on top of a large pretrained encoder.
These papers motivate the proposal through one recurring fact: within-family accuracy is saturated, while unseen-family accuracy is not. NTFold reaches F1 0.977 on RNAStrAlign but 0.645 on the unseen-family split bpRNA-TS0 (NTFold).
PriFold is the closest prior work, and the main contrast. It already augments with covariation. As far as the file describes, though, it treats the augmented sequences as interchangeable and does not ask which variant to train on, or how strongly to weight it.
The language side supplies the method. Training Prompt Matters (SAPO) shows that equivalent prompts cause up to a 156% difference in forgetting and generalization. It also shows that low pre-update loss predicts lower gradient conflict, meaning the update on the new example pushes the weights less directly against the rest of training. Its method, SAPO, selects among about 20 LLM-generated paraphrases. SAPO has never been applied outside natural-language instructions, and it depends on an external paraphrasing model. RNA removes that dependency, because structure-preserving variants can be generated exactly from the structure itself.
The multi-objective literature constrains the design. Fixed loss weights are brittle across datasets (Domain Separation Networks; Generating Images with Perceptual Similarity Metrics based on Deep Networks). Gradient surgery, which projects out the conflicting parts of two losses' gradients, is the standard remedy (Gradient Surgery for Safe LLM Fine-Tuning; Conflict-Averse Gradient Aggregation). It costs extra gradient computation and memory per objective. We test whether choosing better training inputs removes the conflict more cheaply than reshaping gradients.
Core Claim
During training, take an example $x$ with true structure $y$. Draw a pool $V(x)$ of variants that preserve $y$, grouped by edit distance so the comparison is not trivially won by near-copies. Pick the variant the current model with weights $\theta_t$ finds easiest:
$$x^* = \arg\min_{x' \in V_d(x)} \ell(f_{\theta_t}(x'), y),$$
Here $V_d(x)$ is the set of variants at mutation distance $d$, and $\ell$ is the backbone's contact-map loss. Then train on
$$\mathcal{L} = \ell(f_\theta(x), y) + \lambda\, \ell(f_\theta(x^*), y) + \mu\, D(f_\theta(x), f_\theta(x^*)).$$
In this objective, $\lambda$ weights supervision on the variant, and $\mu$ weights a consistency term $D$. $D$ is the mean squared difference between the two predicted pair-probability maps.
We claim three things relative to random variant sampling at matched compute:
- Selection improves pseudoknot F1 on unseen families.
- It raises the cosine between the gradients of the original-example loss and the variant loss throughout training.
- It widens the range of $\lambda$ and $\mu$ over which unseen-family accuracy stays near its best.
The third claim is the multi-objective payoff. If variants agree with the main objective, the balancing weight should matter less.
Research Program
Variant generation. Variants come from two sources. The first is real homologs: sequences within an RNAStrAlign family that share an aligned structure. These are natural paraphrases but are limited in number. The second is synthetic compensatory mutations. At paired positions we swap among canonical and wobble pairs (G–C, C–G, A–U, U–A, G–U); at unpaired positions we substitute freely.
Synthetic mutants can change the true fold, so the labels may be wrong. To screen for this, we fold the nested part of each mutant with ViennaRNA and discard mutants whose nested fold drifts away from that of the original. This check does not cover the pseudoknotted pairs, and we will report that limitation openly.
We will also mutate pseudoknot stems separately from nested stems. This tests a sub-hypothesis: crossing pairs are where family memorization concentrates.
Backbone. RFold-style row/column factorization is the primary model. It is fast, which matters when every step scores several variants, and it always produces valid structures. We will replicate the main comparison on an NTFold-style backbone, so the result is not specific to one architecture.
Measurements logged during training. We record three things: the gradient cosine between the original and variant losses, the pre-update loss of the chosen variant, and its edit distance. Scoring a pool of K variants costs K forward passes per example. We will rescore only every few steps, and we report wall-clock overhead against SAPO's reported 8–12% figure (SAPO).
Baselines
Naive reference. This is the backbone trained on RNAStrAlign with no augmentation, using its published recipe. It shows how much any form of paraphrasing helps.
Strongest existing method. This is PriFold, which reports RNAStrAlign F1 0.988, pseudoknot F1 0.944, and zero-shot ArchiveII F1 0.952. RFold reports 0.977 overall and 0.982 on pseudoknots (RFold). On unseen families, the only figure in the file is NTFold's bpRNA-TS0 F1 of 0.645 (NTFold). We found no reported result on Lacuna for RFold, PriFold, or DEPfold on bpRNA-new.
For a fair comparison, we will re-implement PriFold-style random covariation augmentation on our backbone. It gets the same variant pool, the same number of training steps, and the same 20-trial tuning budget over $\lambda$ and $\mu$. Comparing against PriFold's full pretrained encoder would confuse the effect of pretraining with the effect of selection. We will also report PriFold's published numbers as an external reference.
What counts as success. Beating this baseline means at least 2 F1 points on bpRNA-TS0 pseudoknotted pairs and on bpRNA-new, averaged over three seeds, without losing more than 0.5 F1 within family.
Ablations that isolate the selection rule. These keep everything else fixed:
- highest-loss selection (hard-example mining);
- selection by gradient cosine instead of loss;
- selection without edit-distance grouping;
- the consistency term $\mu$ switched off;
- random variants trained with PCGrad-style gradient surgery between the two loss terms;
- a static upweighting of crossing pairs in the loss, with no variants at all.
Why not just…?
Why not just augment randomly, as PriFold already does? This is the most direct threat. Random augmentation could plausibly match selection if the variant pool is large and the labels are clean. The prompt-tuning result (SAPO) gives a reason to expect otherwise: in language, random prompts degraded retention where selected ones improved it. But no one has shown this transfers to sequences. The selected-versus-random comparison at matched compute settles it. If random augmentation wins or ties, we will report that SAPO's finding does not carry over to RNA.
Why not pick the hardest variants instead? Hard-example mining is the textbook intuition, and it predicts the opposite of our rule. We include it as an arm. Low-loss selection beating high-loss selection would be the most informative possible result, because it would separate the "align with the model's current state" mechanism from plain "more signal per step."
Why not just use gradient surgery or static weights? Static weights cannot undo gradient components that point in opposite directions (Gradient Surgery for Safe LLM Fine-Tuning). Gradient surgery can, but it needs a separate gradient per objective. Our PCGrad arm and our static crossing-pair arm test both directly. If surgery matches selection at lower cost, selection is not worth it. If static crossing-pair weighting alone closes the gap, the problem was class imbalance, not memorization.
Why not instruction-tune a biology LLM? Biology Instructions covers sequence-level tasks only. Its models have no dense pairing output that could represent crossing pairs. Conditioning a folding model on natural-language text adds nothing that a structure-preserving sequence cannot express more exactly.
Proposed Experiments
Data and splits. We train on RNAStrAlign. We evaluate on ArchiveII, bpRNA-TS0, and bpRNA-new, plus a leave-one-family-out protocol on RNAStrAlign.
Metrics. We report:
- F1 and Matthews correlation on all pairs and on crossing pairs separately;
- structure validity;
- the Weisfeiler-Lehman graph-kernel similarity from Rethinking Performance Measures, because F1 drops to zero when a helix shifts by a single position and so overstates some errors;
- a paraphrase-consistency score: agreement of predictions across held-out structure-preserving variants of test sequences, the RNA analogue of a prompt-sensitivity test.
Decisive checks. There are three:
- Low-loss selection beats random and high-loss selection on unseen-family crossing-pair F1.
- The training-time gradient cosine is higher for selected variants and predicts the final gain.
- The accuracy-versus-$\lambda$ curve is flatter under selection.
If the first check fails, the transfer hypothesis is rejected. If the first passes but the second does not, the gain would have to come from a mechanism other than reduced gradient conflict.
References
- ViennaRNA Package 2.0
- Predicting RNA Secondary Structures with Arbitrary Pseudoknots by Maximizing the Number of Stacking Pairs
- RNA Secondary Structure Prediction By Learning Unrolled Algorithms
- Deciphering RNA Secondary Structure Prediction: A Probabilistic K-Rook Matching Perspective
- NTFold: Structure-Sensing Nucleotide Attention Learning for RNA Secondary Structure Prediction
- DEPfold: RNA Secondary Structure Prediction as Dependency Parsing.
- PriFold: Biological Priors Improve RNA Secondary Structure Predictions
- Training Prompt Matters: State-Adaptive Optimization for Robust Fine-Tuning
- Domain Separation Networks
- Generating Images with Perceptual Similarity Metrics based on Deep Networks
- Gradient Surgery for Safe LLM Fine-Tuning
- Conflict-Averse Gradient Aggregation for Constrained Multi-Objective Reinforcement Learning
- Biology Instructions: A Dataset and Benchmark for Multi-Omics Sequence Understanding Capability of Large Language Models
- Rethinking Performance Measures of RNA Secondary Structure Problems