When a long non-coding RNA (lncRNA) — an RNA molecule transcribed from the genome that does not code for protein — binds a messenger RNA (mRNA), the duplex can end in opposite fates. SNHG5 binds the 3' untranslated region (UTR) of the SPATS2 transcript and physically blocks the decay-promoting protein STAU1, extending the mRNA's lifetime (SNHG5 promotes colorectal cancer cell survival by counteracting STAU1-mediated mRNA destabilization). Other duplexes on the same kind of UTR motifs recruit the decay machinery and shorten it. Current computational tools cannot tell these two outcomes apart: thermodynamic predictors report binding energy but not consequence, and every RNA language model in the literature models only a single molecule at a time. This proposal builds a transformer that reads a lncRNA and an mRNA UTR as two interacting sequences and predicts not just where they hybridize but whether the hybridization shields the mRNA from degradation or promotes it.
The proposal sits at a genuine three-way intersection. The biology supplies the mechanism (trans-acting lncRNAs controlling mRNA half-lives through UTR occupancy); the transformer architecture supplies the representation (cross-attention between two sequences, with path length of one between any lncRNA position and any UTR position); and the interdisciplinary NLP tradition supplies the methodological lens — specifically, the injection of symbolic domain knowledge into attention (as sememes are injected into Chinese definition modeling in Incorporating Sememes into Chinese Definition Modeling and base-pairing rules into attention in ERNIE-RNA: an RNA language model with structure-enhanced representations) and white-box interpretability testing (as in White-box Testing of NLP models with Mask Neuron Coverage). The reader should be able to carry away: we extend single-sequence RNA transformers to two-sequence pairs, and we ask them to predict a biological function — stabilization versus destabilization — that no model has been asked to predict before.
Core Claim
The central claim is that intermolecular base-pairing compatibility between a lncRNA and an mRNA UTR, injected as a pairwise bias into a cross-attention layer, lets a modest transformer trained on interaction data predict the functional fate of an RNA duplex — stabilization, destabilization, or neutrality — with accuracy that no thermodynamic tool or single-sequence language model can reach, because neither class of model represents the competition between the lncRNA and decay factors for the same UTR motif at all. The key biological observation motivating this comes from Damas et al. (SNHG5): STAU1 binding to the SPATS2 3' UTR promotes decay, while SNHG5 binding to that exact region displaces STAU1 and stabilizes the transcript. Fate depends on competitive occupancy, which is a relational property of two sequences plus context — precisely what cross-attention can represent and what per-sequence pipelines cannot.
Research Program
Three components, in order of construction. First, a two-tower encoder: one tower embeds the lncRNA (or a candidate segment of it), the other embeds the UTR, both initialized from a pre-trained RNA language model of the ERNIE-RNA type, which was pre-trained on 20.4 million non-coding RNA sequences and already encodes intramolecular structure in its attention (ERNIE-RNA). The two towers meet in a cross-attention layer whose query vectors come from lncRNA tokens and key/value vectors from UTR tokens, augmented with an intermolecular base-pairing bias:
$$\text{Cross-Attn}(X,Y) = \text{softmax}\!\left(\frac{QK^\top}{\sqrt{d}} + \beta B\right)V$$
where $X$ and $Y$ are the lncRNA and UTR token sequences, $Q$, $K$, $V$ are their linear projections, $B_{ij}$ scores Watson–Crick and GU compatibility between lncRNA position $i$ and UTR position $j$ (analogous to ERNIE-RNA's intramolecular pairwise bias, but across molecules), and $\beta$ is a learned scalar. Readout heads predict three outputs: per-nucleotide binding probability on the UTR, a three-way fate label (stabilize, destabilize, neutral), and, where data permit, a regression on the change in mRNA half-life.
Second, a candidate-generation front end. Full lncRNAs run to thousands of nucleotides and joint attention scales as $O(L^2)$, so exhaustive pairing is intractable. We reuse the seed-and-extension logic of RIblast: An ultrafast RNA-RNA interaction prediction system for comprehensive lncRNA interaction analysis — which pre-computes accessibility energies and finds hybridization seeds via suffix arrays, running 64 to over 250 times faster than exhaustive tools like IntaRNA — to propose candidate duplex windows, and the cross-attention transformer then re-scores each window and assigns its fate. This hybrid also gives a pretraining signal: RIblast energy scores serve as weak supervision for the binding head before any fate labels are seen, distilling thermodynamics into learned representations rather than replacing them.
Third, an interpretability audit in the MNCOVER style (White-box Testing of NLP models with Mask Neuron Coverage): mask individual cross-attention heads and nucleotide positions and measure whether the fate prediction flips. The decisive mechanistic check is whether, on the SPATS2 case, the cross-attention map localizes on the STAU1-displacement motif documented by Damas et al., and whether masking that motif — and only it — reverts the predicted fate toward destabilization.
Proposed Experiments
Evaluation runs on three tiers. Tier one is target recovery on interactome data: the SNHG5 RIA-seq dataset of 121 cytoplasmic target transcripts from Damas et al. (SNHG5) and the TINCR human lncRNA interaction dataset used to validate RIblast (RIblast). Metrics are AUROC and AUPRC for discriminating bound from unbound transcripts, and per-nucleotide precision/recall against the reported binding regions. Tier two is fate classification on a curated benchmark we assemble from the literature: documented stabilizing pairs — SNHG5–SPATS2, lncTASR–AXL 5' UTR (Targeting the TR4 nuclear receptor-mediated lncTASR/AXL signaling with tretinoin increases the sunitinib sensitivity to better suppress the RCC progression), LINC00355–GEFT with LIN28A (LINC00355), MIR205HG–IL-33 (Involvement of lncRNA MIR205HG in idiopathic pulmonary fibrosis and IL-33 regulation via Alu elements), SERTAD4-AS1–SERTAD4 (SERTAD4-AS1 suppresses pancreatic cancer progression by stabilizing SERTAD4 and inhibiting the Notch1 pathway), and the MIPRL cardiac targets (The ischemia-enhanced myocardial infarction protection-related lncRNA protects against acute myocardial infarction) — as stabilizing positives, with negatives drawn from RIA-seq-bound transcripts that show no half-life change in the corresponding Actinomycin D chase, plus destabilizing STAU1-recruiting contexts. We evaluate with leave-one-lncRNA-out cross-validation so the model must generalize to lncRNAs it has never seen, and we report confidence intervals appropriate to the small labeled set. Tier three is the mechanistic audit on the SPATS2 and AXL cases described above. For tier-two regression on half-life change at transcriptome scale, no reported result found on Lacuna for any method predicting lncRNA-mediated half-life change, which is itself the gap this proposal targets; the nearest quantitative anchors come from adjacent tasks and are used only to calibrate expectations (mRNA2vec's Spearman 0.53 on coding-sequence stability, ERNIE-RNA's fine-tuned F1 of 0.873 on bpRNA-1m secondary structure).
Baselines
The naive reference point is chance performance on shuffled lncRNA–UTR pairs, which fixes AUROC at 0.5 and fate accuracy at the majority-class rate; any real signal must clear this. The strongest existing competitor for the binding sub-task is RIblast (RIblast), the fastest intermolecular interaction predictor in the file, validated on the same TINCR human lncRNA data we use; we will run it ourselves on identical candidate sets and data splits, since no reported accuracy figure for RIblast on human lncRNA–mRNA target discrimination appears in the file (only its speedups of 64x to 250x over IntaRNA and RNAplex). Beating RIblast means higher AUROC/AUPRC on target recovery plus any non-trivial fate accuracy at all, since RIblast outputs only energy scores and cannot be scored on fate without an arbitrary energy threshold we would have to tune. A second strong competitor is a Random Forest or gradient-boosted model on engineered features — k-mer composition, GC content, minimum free energy, and RIblast scores — which is exactly the tabular baseline that Suga et al. tested on a related UTR-pairing task, where a Random Forest reached only about 0.55 AUROC while deep representations reached 0.726 in humans (Deciphering the comprehensive relationship between 5′UTR and 3′UTR sequences with deep learning); we grant it the same feature budget, tuning budget, and folds as our model. Finally, the ablation that isolates our key ingredient: the identical two-tower transformer with the intermolecular bias $B$ removed and the two sequences simply concatenated into one input, so any gain is attributable to cross-attention with base-pairing structure rather than to parameters or pretraining. All models receive the same candidate windows, the same pretraining corpus, and the same hyperparameter search budget.
Why not just…?
Why not just use thermodynamic folding tools? RNAfold-class methods are the incumbent, and on intramolecular secondary structure they remain respectable — RNAfold reached a median F1 of 0.492 on bpRNA-1m versus 0.552 for ERNIE-RNA's zero-shot attention (ERNIE-RNA). But on degradation-relevant outcomes they plateau: on the PERSIST-seq full-length mRNA benchmark, RNAfold achieved a Spearman correlation of only 0.25, against 0.36 for the linear DegScore baseline and 0.48 for the deep Kazuki2 model (Predictive models of RNA degradation through dual crowdsourcing). More fundamentally, thermodynamics has no representation of who else is bound: the SNHG5/STAU1 competition on the same SPATS2 motif is invisible to an energy function over two sequences. Our design keeps thermodynamics in the loop (RIblast proposes candidates and supervises the binding head) rather than discarding it, so the comparison in our experiments is additive, not adversarial.
Why not just fine-tune an existing single-sequence RNA language model on the pair? This is the most serious objection, and we test it directly as our concatenation ablation rather than arguing around it. Existing models give it surface plausibility: ERNIE-RNA already predicts structure from attention, and mRNA2vec already unifies segments across a junction, reaching a Spearman of 0.53 on coding-sequence stability versus CodonBERT's 0.34 (mRNA2vec: mRNA Embedding with Language Model in the 5'UTR-CDS for mRNA Design). But mRNA2vec excludes the 3' UTR — the principal lncRNA binding site (SNHG5; LINC00355) — and Equi-mRNA is mathematically restricted to coding triplets, so it cannot encode lncRNAs or UTRs at all (Equi-mRNA: Protein Translation Equivariant Encoding for mRNA Language Models). If concatenation matches the biased cross-attention on fate classification, the honest conclusion is that the bias is unnecessary and the contribution shrinks to the dataset and task; we will report that outcome if it occurs. We note, though, that concatenation gives the model no positional hint of which lncRNA nucleotide faces which UTR nucleotide, whereas the bias encodes exactly that facing relation.
Why not just use classical machine learning on sequence features? Random Forests and SVMs are cheap and sometimes sufficient; on peptide aggregation, an MLP reached an $R^2$ of 0.920 against a transformer's 0.944, a small gap (Efficient Prediction of Peptide Self-assembly through Sequential and Graphical Encoding). But on relational RNA tasks the gap is large: the Random Forest baseline scored roughly 0.55 AUROC — near chance — on coordinated UTR pairing, where foundation-model representations with contrastive learning scored 0.726 (Suga et al.), and the linear DegScore degradation model was beaten by 37% on the OpenVaccine public benchmark (OpenVaccine). With our small labeled set, overfitting the transformer is a real risk, which is why the feature model is run as a full baseline with equal tuning budget and why we lean on pretraining and leave-one-lncRNA-out validation to expose it.
Why not just extend LinearDesign's grammar-based parsing? LinearDesign showed that formal-linguistic machinery — a stochastic context-free grammar intersected with a word lattice via beam search — can optimize mRNA stability, raising in vitro half-life from 3.9 to 20 hours and producing a 128-fold higher antibody response in mice (Algorithm for optimized mRNA design improves stability and immunogenicity). It is a design tool for a single coding sequence, however, with no notion of UTRs, other molecules, or cellular decay factors. We take from it the interdisciplinary lesson — symbolic structure can be injected into sequence search — and apply that lesson inside attention rather than inside a parser; a grammar-driven design mode for protective lncRNA mimics is the natural follow-up once fate prediction works.
Decisive Checks
The proposal succeeds if three results hold: the cross-attention model beats RIblast on target recovery on the SNHG5 and TINCR interactomes under matched conditions; it classifies fate above the majority-class rate under leave-one-lncRNA-out validation, with the concatenation ablation showing a measurable deficit that attributes the gain to the intermolecular bias; and the masking audit localizes the model's decision to the STAU1-displacement motif on the SPATS2 3' UTR. If the concatenation ablation ties, or if the Random Forest on thermodynamic features matches the transformer on the small fate set, the fallback contribution is the curated fate benchmark itself and the negative result — still useful, since the field currently has no measurement of this question at all. Protein-stabilization mechanisms (FOXP4-AS1 bridging USP7 to ZEB1, circRNA FEACR binding NAMPT) are explicitly out of scope for this round; they extend the fate taxonomy from transcript stability to protein half-life and mark the next iteration.
References
- SNHG5 promotes colorectal cancer cell survival by counteracting STAU1-mediated mRNA destabilization
- Incorporating Sememes into Chinese Definition Modeling
- ERNIE-RNA: an RNA language model with structure-enhanced representations
- White-box Testing of NLP models with Mask Neuron Coverage
- RIblast: An ultrafast RNA-RNA interaction prediction system for comprehensive lncRNA interaction analysis
- Targeting the TR4 nuclear receptor-mediated lncTASR/AXL signaling with tretinoin increases the sunitinib sensitivity to better suppress the RCC progression
- Long Non-Coding RNA LINC00355 Promotes the Development and Progression of Colorectal Cancer by Elevating Guanine Nucleotide Exchange Factor T Expression via RNA Binding Protein lin-28 Homolog A
- Involvement of lncRNA MIR205HG in idiopathic pulmonary fibrosis and IL-33 regulation via Alu elements
- SERTAD4-AS1 suppresses pancreatic cancer progression by stabilizing SERTAD4 and inhibiting the Notch1 pathway
- The ischemia‐enhanced myocardial infarction protection‐related lncRNA protects against acute myocardial infarction
- Deciphering the comprehensive relationship between 5′UTR and 3′UTR sequences with deep learning
- Predictive models of RNA degradation through dual crowdsourcing
- mRNA2vec: mRNA Embedding with Language Model in the 5'UTR-CDS for mRNA Design
- Equi-mRNA: Protein Translation Equivariant Encoding for mRNA Language Models
- Efficient prediction of peptide self-assembly through sequential and graphical encoding
- Algorithm for optimized mRNA design improves stability and immunogenicity