RNA inverse folding — designing a nucleotide sequence that folds into a chosen 3D shape — is bottlenecked by two facts. First, the supervision is tiny: solved RNA structures make up less than 1% of public structural repositories, and the whole transcriptome yields only a few dozen non-redundant structural families, so models trained directly on coordinates overfit and collapse into common motifs (Shen et al.). Second, the best geometric design systems still write the sequence like a sentence: gRNAde: Geometric Deep Learning for 3D RNA inverse design decodes nucleotides causally from the 5' end to the 3' end, which forces the model to commit to early nucleotides before it has seen the distant partners they must pair with — a base at position 5 routinely pairs with a base at position 200.
Both problems have been solved separately, in fields that rarely talk to each other. In code intelligence, where the same pathology — flattening a graph-structured object into a one-directional token stream — was diagnosed years ago, GraphCodeBERT: Pre-training Code Representations with Data Flow showed that injecting the data-flow graph (which variable's value feeds which operation) into a bidirectional encoder via graph-guided masked attention recovers the semantics that sequential reading loses. In sequence modeling, DeltaFlow: Noise-Adaptive Bidirectional Gated Delta Networks for Embedded Language Flows showed that iterative denoising generators are inherently non-causal and can be run with forward-plus-backward linear recurrence at a 2.72× throughput gain over full attention. And in RNA modeling itself, RiboSphere: Learning Unified and Efficient Representations of RNA Structures showed that compressing noisy 3D coordinates into a small vocabulary of discrete structural tokens fights scarcity by exposing a reusable "syntax" of hairpins and internal loops.
This proposal combines the three into one system. The claim: treat the base-pairing network of a target RNA as its program — the direct analogue of a code data-flow graph — and design sequences with a bidirectional, linear-time denoising decoder that conditions every nucleotide on that program, so that no position is ever chosen blind to its pairing partners.
Core Claim
An RNA's secondary and tertiary base-pairing topology is a dependency graph in exactly the sense a program's data-flow graph is: nucleotide $i$'s identity is constrained by its graph neighbors, not by its linear neighbors. The failure modes documented across the three literatures — gRNAde's causal 5'-to-3' commitment, the "last-pair problem" (chronic near-miss base pairs) reported by Order-Agnostic Decoding for Sample-Efficient RNA Inverse Folding, and the superficial-artifact memorization of sequential models shown in code by Saikat Chakraborty et al. and Hongyu Zhang et al. — are all symptoms of reading a graph as a one-way stream. The proposal is to build PairFlow: an inverse-folding model in which (i) the target structure is compiled into a pair-flow graph — base pairs, backbone adjacency, and tertiary contacts as directed edges, deliberately abstracting away coordinate noise, the way data-flow graphs abstract away syntactic noise — plus a small set of discrete structural tokens from a Finite Scalar Quantization tokenizer in the style of RiboSphere; and (ii) the sequence is generated not token-by-token but by a flow-matching process — a generator that learns to gradually turn random continuous vectors into a valid sequence — realized with bidirectional gated linear recurrence in the style of DeltaFlow, so every position is refined against the whole pair-flow graph at every denoising step, in $O(L)$ time per step rather than $O(L^2)$.
No existing work occupies this spot. The three closest systems each hold one or two of the three ingredients and drop the third. Order-Agnostic Decoding is bidirectional but works from flat 2D contact matrices with quadratic attention and still suffers its own last-pair problem, because discrete token-by-token decisions still commit. BAnG: Bidirectional Anchored Generation for Conditional RNA Design generates outward from a manually specified anchor with quadratic Invariant Point Attention. gRNAde has rich 3D graph structure but a strictly causal decoder. RiboSphere has discrete structural tokens but standard, non-bidirectional decoders. DeltaFlow and Bidirectional Linear Recurrent Models for Sequence-Level Multisource Fusion (BLUR) prove the linear bidirectional machinery but have never touched a biopolymer or an explicit dependency graph. PairFlow is the union: graph-guided bidirectional attention over the pair-flow graph (from the code side), discrete structural tokens as the scarcity-robust conditioning signal (from the RNA side), and noise-adaptive bidirectional linear recurrence as the generator (from the sequence-modeling side).
Research Program
The work has three components, each a direct transfer with one domain-specific twist.
First, the pair-flow graph compiler. From a target structure (or a designed backbone), extract a directed multigraph whose nodes are sequence positions and whose edges are Watson-Crick and wobble pairs, non-canonical tertiary contacts, and backbone adjacency — the RNA analogue of the code property graph that combines syntax and data flow. The deliberate design choice, following the lesson GraphCodeBERT learned when it abandoned cluttered syntax trees for compact data-flow graphs, is that the pair-flow graph carries operational pairing logic and discards Euclidean coordinate detail, which is dominated by thermal "wobble" — the conformational noise that Su et al. had to explicitly model as thermodynamic ensembles. A graph that ignores wobble is a cheaper, more transferable conditioning signal than raw coordinates, which is precisely what a regime with a few dozen structural families needs.
Second, discrete structural conditioning. A shallow geometric encoder quantizes coarse-grained local frames of the target into discrete tokens via Finite Scalar Quantization, following RiboSphere, which showed this prevents continuous-latent overfitting and yields tokens that map to interpretable motifs. The pair-flow graph and these tokens together form the conditioning context $c$. Because both are discrete and compact, they can be pre-trained on the ~12,000 RNA structures that gRNAde used from RNAsolo, and further bootstrapped with pseudo-labels on uncharacterized sequences in the self-distillation style of RhoFold+ — attacking scarcity from the sequence side where the data is abundant.
Third, the bidirectional linear flow decoder. The sequence lives as continuous per-position embeddings $z_t \in \mathbb{R}^{L \times d}$ at flow time $t \in [0,1]$, initialized as noise and denoised toward a one-hot nucleotide sequence. The flow velocity is computed from three fused signals:
$$v_\theta(z_t, t) = f\big(\,h_t,\; s^{+}_t \oplus s^{-}_t,\; \mathrm{Attn}_{M}(h_t,\, c)\,\big)$$
where $h_t$ is a per-position projection, $s^{+}_t$ and $s^{-}_t$ are forward and backward recurrent states from gated delta units scanned in both directions (parallel or alternating across layers, per DeltaFlow), and $\mathrm{Attn}_{M}$ is attention restricted by the pair-flow graph mask $M$ — each position attends densely to its graph neighbors and to the structural tokens, sparsely to everything else, exactly as GraphCodeBERT restricted variable nodes to their data-flow relatives. Decay and write rates of the recurrent memory are modulated by the flow timestep (noise-adaptive memory control), which DeltaFlow showed prevents the entropy collapse of unidirectional recurrence. Two design constraints come from the RNA side: the pre-trained RNA language model RNA-FM is kept frozen and injected through lightweight adapters, because EVO-RDesign: Leveraging Evolutionary Priors for Structure-Based RNA Design found that fine-tuning pre-trained models on scarce structures degrades performance; and the training objective adds a mild negative-design term — sequences sampled from intermediate flow states are penalized if a folding oracle finds them more stable in a wrong conformation — addressing a gap gRNAde itself flags as absent.
The scientific questions the program answers are: does graph-masked bidirectional refinement remove the early-commitment failure (measured by near-miss pair rates and perfect-design rates on pseudoknotted targets), does discrete-plus-graph conditioning beat coordinate conditioning under scarcity (measured by recovery on held-out families), and does the linear-time decoder hold up against quadratic bidirectional attention at the sequence lengths where RNA design actually fails (gRNAde reports recovery degradation at length; EVO-RDesign reports the same for long RNAs).
Proposed Experiments
The primary benchmark suite is the one the RNA design field already uses, so results are directly comparable. On RNAsolo sequence recovery, the target is to beat the strongest reported number, the 63.0% recovery of RiboSphere, with results stratified by sequence length to test the long-sequence claim. On the Das et al. (2010) benchmark of 14 target structures, the reference points are gRNAde's 56% native sequence recovery (versus Rosetta's 45%) and RiboSphere's 1.25 Å RMSD / 0.84 TM-score on Das-derived reconstruction clusters. On Eterna OpenKnot Round 7b — 240-nucleotide pseudoknotted designs, the hardest public test of exactly the long-range-commitment failure this proposal targets — the number to beat is the 33.24% perfect-design rate of Order-Agnostic Decoding, which itself is a 5× improvement over the 6.64% of the best published autoregressive protocol; that 6.64% is the cleanest published quantification of the causal bottleneck and the core motivation for the decoder swap. On Eterna100, the bar is 64/100 puzzles solved (Order-Agnostic) against Struct2Seq's 50/100. Generalization to unseen families is measured on the Rfam and RNA-Puzzles zero-shot splits, where EVO-RDesign reports +4.1% and +5.1% sequence recovery gains respectively over its structure-only counterpart. Designed sequences are refolded with a forward predictor in the RhoFold+ family (which reports 4.02 Å average RMSD on 24 RNA-Puzzles targets) and scored with the ViennaRNA / RibonanzaNet-SS oracles, with the in-silico oracle bias treated as a stated limitation; the decisive endgame is submitting designs to a future Eterna wet-lab round, where gRNAde's 50% synthesis success rate (versus Rosetta's 35%) in OpenKnot Round 6 sets the physical bar. A separate efficiency track measures wall-clock and memory versus a parameter-matched quadratic bidirectional diffusion decoder at $L = 100, 240, 500$, where the flow-based RNA generators in the RiboFlow / RiboGen family pay $O(KL^2)$ and BAnG reports being roughly 50× faster than diffusion — the linear decoder should extend that curve, and DeltaFlow's 2.72× throughput gain over full attention at 16k-token lengths says the machinery scales.
Baselines
The naive reference point is Rosetta/FARNA-style physics-based design plus a purely causal neural decoder. Rosetta is the classic energy-minimization pipeline; gRNAde reports it achieving 45% native sequence recovery on the Das benchmark and a 35% wet-lab success rate on OpenKnot Round 6, so it anchors what physics alone does. The causal neural reference is gRNAde itself (gRNAde: Geometric Deep Learning for 3D RNA inverse design) — an SE(3)-equivariant graph network over 3-bead RNA backbones with a left-to-right decoder, reporting 56% native recovery on Das and trained on 12,000+ RNAsolo structures; it is the strongest system that shares this proposal's graph-structural conditioning but not its bidirectionality, so beating it on OpenKnot-style pseudoknots while matching it on Das isolates the decoder claim. The strongest overall competitor is the order-agnostic Transformer of Order-Agnostic Decoding for Sample-Efficient RNA Inverse Folding, with 33.24% perfect designs on OpenKnot 7b and 64/100 on Eterna100; it shares the non-causal insight but not the graph conditioning, discrete tokens, or linear-time decoder, so it is the decisive head-to-head. A third comparator is RiboSphere at 63.0% RNAsolo recovery, which shares the discrete-token ingredient but not the bidirectional flow decoder. All systems will be run on identical target splits, with parameter counts matched within 10%, the same training corpus (RNAsolo structures plus the same RNAcentral pre-training pool), the same sampling budget of 100 sequences per target as gRNAde, the same oracle scoring, and the same tuning budget (one sweep of learning rate and sampling steps each). Beating the baseline means: strictly higher perfect-design rate than 33.24% on OpenKnot 7b, higher recovery than 63.0% on RNAsolo, and no regression on Das. The key ablation runs the identical PairFlow stack with the backward scan deleted — a unidirectional gated-delta flow decoder over the same pair-flow graph and tokens — which isolates bidirectionality as the active ingredient; a second ablation replaces the graph-guided attention mask with dense attention, isolating the structural ingredient; a third swaps discrete structural tokens for raw coordinates, isolating the scarcity ingredient.
Why not just…?
Why not just use an order-agnostic autoregressive Transformer, since it already works? This is the most serious objection, and the honest answer is that on easy targets it may be enough: Order-Agnostic Decoding already showed a 5× reliability gain on OpenKnot 7b. But it has two published failure modes PairFlow is built around. It still decodes token-by-token, so near the end of a permutation the last positions are forced with dwindling options — the "last-pair problem" the authors themselves report — whereas iterative flow refinement lets every position revise jointly against the full graph at every step. And its attention is quadratic, which is the binding constraint precisely on the long pseudoknotted targets where causal methods collapse. The head-to-head at matched parameters and the near-miss-pair diagnostic (fraction of designed pairs off by one position) will settle whether the difference is real; if the order-agnostic model matches PairFlow on OpenKnot 7b and Eterna100, the proposal's contribution reduces to an efficiency result, and we will say so.
Why not just fix gRNAde — keep its 3D graph encoder and change the decoding order? Because the failure is architectural, not ordering alone: gRNAde conditions on noisy Euclidean coordinates that scarce-data regimes overfit, and its authors' own wet-lab round showed half of designs failing physically. Swapping decoding order inside gRNAde is exactly the "delete the backward scan / reorder" ablation family above, and we will run it; but the scarcity argument for discrete structural tokens comes from RiboSphere's finding that quantized tokens prevent latent overfitting where coordinates do not, and EVO-RDesign's finding that long-sequence performance degrades under continuous prior injection. If the gRNAde-plus-reordering ablation matches full PairFlow on RNAsolo and Rfam zero-shot, the tokenization claim dies and the decoder claim survives — that outcome would itself be informative, and the ablation is designed to detect it cheaply.
Why not just scale a sequence language model on RNAcentral, or self-distill like RhoFold+? Abundant sequences do not add structural supervision — the imbalance is structural, not sequential — and the code literature is a controlled warning about what pure sequence scaling buys: Saikat Chakraborty et al. and Hongyu Zhang et al. showed sequential models hitting high benchmark numbers through artifact and keyword memorization rather than learned logic. Self-distillation (RhoFold+'s pseudo-labeling of 27,000+ uncharacterized sequences) is compatible with PairFlow and is part of the plan for the token pre-training stage; it is a complement, not a competitor. The residual risk is expressivity: DeltaFlow reports a 2.78 BLEU deficit for linear bidirectional recurrence versus full attention on hard conditional generation. The pair-flow attention mask is the mitigation — the graph tells the model exactly which long-range dependencies matter, replacing the dense attention whose cost caused that gap — and the dense-versus-masked ablation on RNA tasks directly measures whether the mitigation holds outside text.
Decisive Checks
The proposal lives or dies on three numbers. If PairFlow's perfect-design rate on OpenKnot Round 7b does not exceed 33.24%, the bidirectional-flow thesis is wrong for this task. If its near-miss pair fraction is not substantially below the order-agnostic baseline's, the last-pair problem was misdiagnosed as a commitment problem rather than a refinement problem. And if the unidirectional ablation — same graph, same tokens, backward scan deleted — performs as well as the full model, then the RNA side never needed bidirectionality at all, and the interesting result would be that graph-guided discrete conditioning alone explains the gains. Each of these outcomes is publishable as a clear finding, which is the property a one-deep proposal should have.
References
- Accurate RNA 3D structure prediction using a language model-based deep learning approach
- gRNAde: Geometric Deep Learning for 3D RNA inverse design
- GraphCodeBERT: Pre-training Code Representations with Data Flow
- DeltaFlow: Noise-Adaptive Bidirectional Gated Delta Networks for Embedded Language Flows
- RiboSphere: Learning Unified and Efficient Representations of RNA Structures
- Order-Agnostic Decoding for Sample-Efficient RNA Inverse Folding
- Deep Learning Based Vulnerability Detection: Are We There Yet?
- Diet Code is Healthy: Simplifying Programs for Pre-Trained Models of Code
- BAnG: Bidirectional Anchored Generation for Conditional RNA Design
- Bidirectional linear recurrent models for sequence-level multisource fusion
- Integrating thermodynamic and sequence contexts improves protein-RNA binding prediction
- EVO-RDesign: Leveraging Evolutionary Priors for Structure-Based RNA Design