Isoform-Aware Contrastive Margins: Letting Splice Variants Diverge Only Where Their Structure Says They Should

RNA foundation models learn a numeric summary, or embedding, of each transcript by contrastive training. Contrastive training pulls together sequences that should count as "the same thing" and pushes apart unrelated ones. The strongest such model in our file, Orthrus, treats alternative splice isoforms of one gene as "the same thing." Alternative splicing cuts one gene's RNA into several different mature transcripts, called isoforms. The trouble is biological. Isoforms of one gene can do opposite jobs: an anti-apoptotic variant of a pro-apoptotic gene, a soluble decoy receptor against a membrane-bound signaling form, a constitutively active truncated androgen receptor. A loss that pulls every isoform pair together pays the model to erase exactly these differences. The Orthrus authors flag this risk themselves.

We propose a simple change. Each isoform gets a budget for how far it may drift from its gene's average embedding. The budgets come from one small optimization per gene. That optimization is the edge-weighted vertex cover used to build search heuristics in multi-agent path finding. Pairs of isoforms that differ structurally in ways known to matter get a required separation. The vertex cover then decides which isoform should carry that separation, and it avoids charging the same difference twice. All other isoforms stay pulled toward the gene centroid, so the benefit of treating isoforms as related is kept.

Core Claim

Our claim is that the contrastive invariance problem has the same structure as a covering problem, and that solving that covering problem gives a better contrastive target than either full invariance or dropping isoform pairs altogether. Suppose every isoform $u$ of gene $g$ sits at distance $d_u = \lVert z_u - m_g \rVert$ from the gene centroid $m_g$, where $z_u$ is the embedding of $u$. By the triangle inequality, two isoforms $u, v$ can be separated by at least $w_{uv}$ only if $d_u + d_v \ge w_{uv}$. The smallest total drift from the centroid that still allows every required separation is therefore

$$\min_{c \ge 0} \sum_{u \in I_g} c_u \quad \text{s.t.} \quad c_u + c_v \ge w_{uv} \;\; \forall (u,v) \text{ with } w_{uv} > 0,$$

where $I_g$ is the gene's isoform set and $c_u$ is isoform $u$'s drift budget. This is the edge-weighted minimum vertex cover that Improved Heuristics for Multi-Agent Path Finding with Conflict-Based Search uses to turn pairwise conflict costs into an admissible bound. In that setting, the cover stops one agent's single detour from being counted once per conflict it resolves. In ours, it stops one divergent isoform from forcing every sibling to drift. If one variant retains an intron and differs from four siblings, the cover places the budget on that variant and leaves the siblings near the centroid. Pairwise repulsion alone would push all five apart.

The training loss keeps Orthrus's objective and adds two terms:

$$\mathcal{L} = \mathcal{L}_{\text{Orthrus}} + \lambda \sum_{u} \max(0,\, d_u - \alpha c_u) + \mu \sum_{(u,v)} \max(0,\, \alpha w_{uv} - \lVert z_u - z_v \rVert).$$

The first added term pulls an isoform toward its centroid only beyond its budget. The second keeps flagged pairs apart. The constant $\alpha$ converts divergence units into embedding distance, and $\lambda, \mu$ are loss weights. The isoform-pair positives inside $\mathcal{L}_{\text{Orthrus}}$ are replaced by the centroid term. The orthology positives across species, from Orthrus, are left untouched.

Where the Divergence Weights Come From

Isoform-level functional labels are scarce. A Literature Review of Gene Function Prediction by Modeling Gene Ontology notes that Gene Ontology and CAFA annotations are almost all at the gene level. For that reason, $w_{uv}$ must come from annotation structure alone, never from labels or from any downstream model we later evaluate against.

We borrow a lesson from topic segmentation, the task of cutting a dialogue into topically coherent chunks. Work there finds that the unit of comparison matters a great deal. SeCom reports that topic-sized memory units beat both single turns and whole sessions. Accordingly, we compare isoforms segment by segment rather than as whole sequences. Segments are the exon-level pieces of the splice graph, grouped into 5′ untranslated region, coding region, and 3′ untranslated region, as RNA-GPS does for its features.

A pair receives weight from the segments that differ between its two members, scored by a small fixed rubric. The rubric covers a frameshift or lost coding exon, a predicted nonsense-mediated decay trigger, a retained intron, and a 3′ untranslated region difference overlapping known splicing-factor binding motifs such as HuR, PTBP1, and TIA1. RNA-GPS identifies these motifs as localization signals and identifies intron retention as a cause of nuclear anchoring. Pairs that differ only by synonymous or very short untranslated-region changes get $w_{uv}=0$.

The heuristic-overhead literature sets one constraint here. The vertex cover is NP-hard in general, and per-node cover solving is a known cost in path finding. In our setting, however, the cover is solved once per gene, offline, over typically fewer than a dozen isoforms. Exact integer programming is therefore cheap. We will report its wall-clock cost so the overhead claim can be checked rather than assumed.

Research Program

We will build the weight rubric over GENCODE annotations, solve the per-gene covers, and continue pretraining from an Orthrus checkpoint under the modified loss. If no checkpoint can be obtained, we will retrain a reduced Orthrus on a fixed subset of the corpus. Every arm uses the same steps, data, and hyperparameter search budget.

The analysis then asks three questions. First, do the learned embeddings separate functionally divergent sibling isoforms? Second, does gene-level transfer survive? Third, is the vertex cover itself responsible for any gain, or would cruder weighting do as well?

Baselines

The do-nothing reference is the unmodified Orthrus objective with isoform pairs as positives, continued for the same number of steps. Its reported anchor is human RNA half-life, where a linear probe reaches Pearson $R = 0.69$, against 0.45 for HyenaDNA (Orthrus). Our method must not fall meaningfully below this number. Beating Orthrus means matching half-life within run-to-run noise while improving on the isoform-resolution tasks below.

The strongest task-specific competitor on localization is the RNA-GPS random forest over segment-wise $k$-mers, which counts short fixed-length sequence fragments in each region. It reports overall AUROC 0.77 and AUPRC 0.49 across eight compartments on APEX-seq, and AUROC 0.85 on nucleus versus cytoplasm against 0.74 for deepLncRNA (RNA-GPS). It is dangerous precisely because it is sequence-local and makes no invariance assumption. We rerun it on our exact gene-held-out splits.

We found no reported result on Lacuna for Orthrus embeddings on APEX-seq localization or on sibling-isoform discordance. Both numbers will be produced here.

Two ablations isolate the key ingredient. The first, "orthology-only," drops isoform positives entirely. The second, "pairwise-margin," keeps the same $w_{uv}$ but applies only the push term, together with a uniform centroid pull with no per-isoform budget. If pairwise-margin ties our method, the vertex cover is not earning its place.

Why not just…?

Why not just fine-tune Orthrus on the downstream task? Fine-tuning can re-learn divergence when labels are plentiful. However, isoform-resolved labels are the scarce resource, and the regime that matters is linear probes and few-shot learning, where Orthrus's appeal lies: it reaches 73% of fully supervised performance with 45 examples. We will report both linear-probe and full fine-tuning results. If fine-tuning closes the gap at small label counts, the proposal's value shrinks to label efficiency, and we will say so.

Why not just drop the isoform positive pairs? This is the real threat. It is simple and removes the erasing pressure at once. What it gives up is the signal that most isoforms of a gene do behave alike. We expect that loss to show up as worse half-life transfer and worse placement of rare isoforms with few orthologs. The orthology-only ablation settles this directly. If it matches us on every task, we will report that the invariance problem is solved by removal and that the covering structure is unnecessary.

Why not just use a $k$-mer random forest, since RNA-GPS already beats deep models? For localization it may well stay competitive. It does not give a reusable representation for half-life or other tasks, however, and it cannot use the evolutionary signal from 400+ mammals. We treat it as the bar on localization rather than something to argue past.

Why not weight each pair directly without the cover? Pairwise weighting charges a divergent isoform's difference to all of its siblings. The pairwise-margin ablation tests whether that matters in practice.

Proposed Experiments

The decisive test is sibling-discordance prediction on the APEX-seq HEK293T atlas, which covers 3,660 transcripts in 8 compartments. For genes with two or more measured isoforms, the task is to predict from the embedding pair whether two siblings localize differently, scored by AUROC on held-out genes. Full Orthrus should do worst on this task. Our method should beat both Orthrus and orthology-only.

A second check tests the mechanism. Among discordant pairs where one member retains an intron, the cover should assign larger $c_u$ to the retained-intron isoform, and that isoform should be the nuclear-enriched one, as RNA-GPS findings predict. This check is falsifiable without any training.

Third, we run standard eight-compartment localization against RNA-GPS, and half-life regression against the 0.69 anchor.

Fourth, as a qualitative probe with honestly small numbers, we rank the embedding distances of documented functional switches against the background distribution of same-gene pair distances. The switches are AR-V7 against full-length androgen receptor (AR splice variants), MEIS2C and MEIS2D against other MEIS2 isoforms (MEIS2C/D), and BIN1 neuronal against microglial isoforms (BIN1).

All arms run three seeds, and we report the per-gene cover solve time alongside training cost. The result that would sink the proposal is orthology-only or pairwise-margin matching it on discordance AUROC within seed variance.