Non-coding RNAs are RNA molecules that do their job through the shape they fold into rather than through a protein they encode. Across species they often keep that shape while their letter-by-letter sequence drifts far apart. To compare them, researchers use structural aligners: programs that line up RNA sequences while also matching the base pairs that hold the fold together. These tools matter most in the "twilight zone," below roughly 55–60% sequence identity, where ordinary sequence alignment stops working.
Two leading aligners, PicXAA-R and LaRA 2, ship with hand-set parameters such as scoring weights, filtering thresholds, and solver step-sizes. Nobody has measured how much those settings matter, or whether they matter differently as sequences diverge. This note proposes a direct measurement and a test of one practical fix. We will measure how much alignment quality could be gained by tuning, as a function of sequence identity. We will then test whether a small table of defaults indexed by identity recovers most of that gain. Identity can be computed cheaply before alignment.
Core Claim
Our hypothesis is that the payoff from tuning a structural aligner grows as sequence identity falls. In other words, the twilight zone is also the regime where hand-set defaults are most likely to be wrong. We further expect the parameters that matter to shift with identity: sequence-scoring terms should dominate at high identity, and structure-weighting terms at low identity.
If this holds, a default table conditioned on identity bin should beat a single "optimal default" chosen across all data. A single global default is the approach that Tunability showed to be valuable for tabular learners. The claim is falsifiable in a useful way. If one global default captures nearly everything, the practical conclusion is just as clear: ship better defaults and stop worrying about tuning.
Grounding in Prior Work
Two papers supply the observation that the right method depends on the identity regime. PicXAA-R builds alignments greedily from base-pairing and alignment probabilities. It beats competing structural aligners below 60% identity on BRAliBase 2.1, and its authors name "automating parameter selection" as open future work. LaRA 2 poses alignment as an integer program solved by Lagrangian relaxation, a method that turns hard constraints into tunable penalties. It reports that structural information becomes critical below 55% identity. Neither paper asks whether the parameters' influence itself depends on identity. That question is our gap.
Two papers supply the measurement method.
- Tunability defines tunability as the gap between a default setting and the best reachable setting. It estimates that gap with surrogate models, which are cheap predictors of performance fitted to logged runs. It reports 0.006 AUC of tunability for random forests and 0.042 AUC for SVMs, plus a 0.045 AUC gain from data-driven defaults for elastic net.
- HyperSHAP splits performance into per-parameter and interaction contributions using Shapley interaction indices. It shows that a "tunability game" picks better parameter subsets than variance-based analysis. It also finds that interactions beyond third order add little, which it calls the "Rule of Three."
We reuse both methods. Our extensions are to condition them on a property of each problem instance (sequence identity), and to test the Rule of Three on a combinatorial aligner where pairing decisions interact through a graph. The latter is an open question in the file's Gap 4.
JAHS-Bench-201 cautions against treating structure and settings separately. It reaches target accuracy up to 33x faster by searching architecture and hyperparameters jointly. Accordingly, we treat the choice between aligners as part of the configuration rather than tuning each one in isolation.
PriFold predicts single-sequence structure with a pretrained transformer and covariation augmentation. It is not an aligner, so it constrains rather than competes. It is a possible source of pairing probabilities in place of RNAfold, and we test that substitution only as an ablation.
Research Program
Configuration space. We will enumerate every exposed parameter of PicXAA-R and LaRA 2. For PicXAA-R these are consistency-transformation weights, edge-filtering limits, and clustering thresholds. For LaRA 2 they are relaxation step-size, iteration limit, convergence threshold, and gap and structure scores. Both tools also share an upstream base-pair probability cutoff from RNAfold. Each parameter gets a range spanning its default by a broad margin.
Running the aligners. Using LaRA 2's reported throughput (about 350,000 pairwise alignments in 26.5 minutes on 16 cores), we can afford tens of thousands of real configuration runs on BRAliBase 2.1. Every run is scored per alignment set.
Surrogate and data split. We fit a random-forest surrogate that maps configuration and identity to quality, following Probst et al. Training and evaluation are split by RNA family, so the defaults learned on some families are always tested on families they never saw.
Tunability and interaction analysis. For each identity bin (above 70%, 60–70%, 50–60%, 40–50%, below 40%), we compute three things:
- Probst-style tunability, meaning best reachable minus default.
- The HyperSHAP tunability game.
- Interaction indices up to fourth order, to check whether the Rule of Three holds.
Defaults and validation. From the training families we derive two candidates: a global optimal default, and a per-bin default table. We then validate both on held-out families with real runs, not surrogate predictions.
Metric choice. The primary metric is Sum-of-Pairs Score (SPS): the fraction of reference-aligned residue pairs that the aligner reproduces. The Structural Conservation Index (SCI) is secondary. SCI rewards a stable consensus fold and can be inflated by settings that force spurious pairs, so tuning against it risks gaming the metric. Any parameter that raises SCI while lowering SPS will be reported as a failure of that metric, not as a gain.
Baselines
Naive reference. Both aligners run at their shipped defaults on the same held-out families, scored by real runs. Every other condition is measured as a gain over this one.
Sequence-only floor. A sequence aligner, MAFFT-xinsi as used in PicXAA-R's comparisons, is run on identical inputs. This shows where structure stops paying for itself. We expect it to stay competitive above about 60% identity.
Strongest existing competitor. This is the Probst-style global optimal default: one configuration per aligner, chosen to maximize mean SPS across training families. It has the same configuration budget and the same family split as our per-bin table. Beating it means the per-bin table improves held-out SPS in the twilight-zone bins by a margin larger than the across-family bootstrap confidence interval, with no loss above 60% identity.
There is no reported result on Lacuna for a tuned or default SPS value of PicXAA-R or LaRA 2 on BRAliBase 2.1. The only nearby figure is PicXAA-R's "6-7% SPS improvement over MXSCARNA" on BraliSub/LocalExtR, so we will regenerate default-setting numbers ourselves.
Oracle ceiling. Per-family Bayesian optimization on the test families defines an upper bound. We report the per-bin table's gain as a fraction of this oracle gain.
Ablations. - Conditioning variable. We replace identity with sequence length, or with random bins of the same size. If either does as well, identity is not the operative variable. - Pairing-probability source. We swap RNAfold probabilities for PriFold's predicted pairings. This tests whether upstream structure quality changes tunability downstream.
Why not just…?
Why not just use the shipped defaults? The aligners' authors validated their defaults on BRAliBase, so they may already sit near the optimum. In some bins this is a real possibility, and the naive baseline will reveal it. However, Tunability found software defaults far from data-driven ones for several learners. PicXAA-R's own authors flag their manual tuning as a limitation. If measured tunability is near zero in every bin, we will report that as the finding.
Why not just learn one better global default? This is the most serious threat. If tunability is flat across identity, a single default captures everything and the per-bin table adds nothing. The head-to-head on held-out families settles this. The proposal deliberately treats "global default suffices" as an acceptable and publishable outcome rather than arguing around it.
Why not just switch to a sequence aligner above 60% identity and use structural defaults below? That rule might capture most of the practical gain. We include it explicitly as a two-regime hybrid baseline. The per-bin table must beat it inside the twilight zone, not only at the boundary.
Why not just use one-at-a-time sensitivity or variance-based importance analysis? HyperSHAP reports that tunability-game subsets beat variance-selected subsets in anytime performance. Randomized Nonmonotone BPGM shows how coupled parameters defeat isolated analysis. We will run variance-based importance alongside the tunability game and check which parameter subset, tuned alone, recovers more oracle gain.
Decisive Checks
Hypothesis 1: tunability rises as identity falls. Bin-wise tunability should rise monotonically as identity drops, and the top-ranked parameters should change between the high and low bins. A flat curve refutes this hypothesis.
Hypothesis 2: identity-conditioned defaults are practical. The per-bin table must beat the global default on held-out families in the bins below 60% identity, while staying within noise above 60%. It must also recover a substantial share of the oracle gain.
Hypothesis 3: interaction structure. If interactions of fourth order or higher carry non-trivial weight in low-identity bins, the Rule of Three fails for structural alignment. That would be a finding in its own right.
Robustness checks. - Every surrogate-based conclusion must replicate with real runs on held-out families. - The tuned settings must transfer from BRAliBase to the Murlet set and to 5SrRNAdb without retuning.
References
- Tunability: Importance of Hyperparameters of Machine Learning Algorithms
- PicXAA-R: Efficient structural alignment of multiple RNA sequences using a greedy approach
- LaRA 2: parallel and vectorized program for sequence-structure alignment of RNA sequences
- HyperSHAP: Shapley Values and Interactions for Explaining Hyperparameter Optimization
- JAHS-Bench-201: A Foundation For Research On Joint Architecture And Hyperparameter Search
- PriFold: Biological Priors Improve RNA Secondary Structure Predictions
- A Randomized Nonmonotone Block Proximal Gradient Method for a Class of Structured Nonlinear Programming