Single-cell multiome experiments measure, in the same cell, which stretches of DNA are open (chromatin accessibility, from ATAC-seq) and how strongly each gene is expressed (RNA-seq). A central biological question is which open regions, called peaks, control which genes. One attractive answer is to train a model in which each gene "looks at" all peaks through cross-attention, a layer where one set of items (genes) computes weighted relevance scores over another set (peaks), and then to read the learned weights as candidate regulatory links. The obstacle is scale. A typical dataset has about 116,000 peaks and 13,000 genes, so full gene-by-peak attention in every cell is expensive. Current models avoid this cost by deciding in advance where a gene is allowed to look. AIM splits the genome into 22 per-chromosome sub-models. SCRIPT masks attention to pairs supported by external Hi-C and eQTL databases. MultiGATE uses a genomic-distance window. Each shortcut rules out, by construction, exactly the links a discovery tool should be able to find: links that are distal, absent from reference databases, or on another chromosome.
We propose to learn the routing itself in two stages. First, a cheap low-dimensional score over every gene-peak pair depends only on gene and peak identity, not on the individual cell. Because it is cell-independent, it is computed once per training step and shared across the whole batch. Second, each gene attends exactly, with cell-specific weights, over only its top-scoring peaks plus a small random exploration set. This keeps the cost near-linear in the number of cells while leaving every pair in the genome eligible. The novelty is not efficiency for its own sake. The claim is that how you sparsify determines what the attention map can later reveal. We will measure directly whether learned routing preserves weak, distal regulatory links that attention-based pruning and fixed windows discard.
Core Claim
Vision work on token pruning already shows that the rule for choosing which tokens to keep shapes what the model can see. FastV prunes visual tokens by early-layer attention mass. AgilePruner shows that attention-only pruning drops isolated low-salience objects, while diversity-based pruning brings in noise and hallucinations. It also notes that it fails on low-complexity scenes that contain many small vital objects. Genomes are the extreme version of that scene: sparse regulatory elements sit inside long, repetitive, mostly irrelevant sequence.
Our claim has three parts. A gene-to-peak cross-attention model with learned, cell-independent routing and a small exploration budget should match distance-windowed and database-masked models on standard cis-regulatory link recovery. It should outperform them on distal links beyond 100 kb. And its attention-derived link rankings should agree with those of an exact, unpruned model far better than rankings from attention-mass pruning do. The third part is the one that matters for interpretation. It tests whether the sparsified attention map is a faithful stand-in for the attention map one would have obtained without sparsification.
Research Program
Architecture. Each peak gets a learned identity embedding $e_p$ and each gene an identity embedding $u_g$. The router scores a pair as $s(g,p) = \langle W_u u_g, W_e e_p \rangle$, with projections into a small dimension $r$ (around 16).
Cost. Computing all scores costs $O(G \cdot P \cdot r)$ once per step, where $G$ is the number of genes and $P$ the number of peaks. This is about $2.5 \times 10^{10}$ multiply-adds, which is modest on one accelerator and does not grow with batch size.
Candidate sets. For each gene we keep the top $k$ peaks by $s$, plus $m$ peaks drawn uniformly from the rest of the genome, resampled every step. The random draw guarantees every pair has nonzero probability of being evaluated and receiving gradient.
Cell-specific attention. Within each cell, queries come from the gene embedding conditioned on a cell-state vector. Keys and values come from the peak embedding modulated by that cell's accessibility at the peak. Exact softmax attention then runs over only the $k+m$ candidates. Per-cell cost is $O(G(k+m)d)$, where $d$ is the attention dimension. The router is trained through the attention loss using a straight-through relaxation of the top-$k$ selection.
Gating. We borrow the learnable gate from Gated Cross Attention, which balances large protein sequences against small drug molecules. Here the gate mixes the peak-derived context with the gene's own state, so that the far larger peak stream cannot wash out gene-level signal. We treat the gate as a supporting component and ablate it rather than making it the headline.
Relation to prior work. This design borrows the "small expensive stream, large cheap stream" logic of LookupViT, which compresses image patches into about 25 global tokens. It deliberately refuses LookupViT's compression. Peaks are never merged into summary tokens, because merging destroys the genomic coordinates that make an attention weight biologically readable. SG-Former similarly aggregates background tokens into coarse ones, which we avoid for the same reason. SToFM uses clustered "virtual cells" for spatial context and also compresses. Cross-Modal Imputation for ST scales dual cross-attention by heuristic cosine-similarity pair sampling. Our routing is instead learned end-to-end and keeps full coordinate resolution.
Link extraction. A gene-peak link score is the attention weight averaged over cells within a cell type. Following the background-subtraction idea of the contrast matrix in the plant RNA foundation model, we subtract the same average computed with accessibility-shuffled cells. This removes weight that a peak receives regardless of its state.
Planted-link simulation. We will build semi-synthetic multiome data by taking real ATAC profiles and generating expression with planted peak-to-gene effects of controlled strength and genomic distance. Planted effects are cis within 50 kb, cis beyond 100 kb, and trans. This gives ground truth for measuring which routing rules lose which links as signal weakens. It is the genomic analogue of AgilePruner's small-object failure case.
Baselines
Naive reference. Per-cell-type Pearson correlation between peak accessibility and gene expression across cells, computed for all pairs. It uses no attention and no model. It is the reference any learned link score must beat to justify itself. No reported result was found on Lacuna for this baseline on the benchmarks below, so we will compute it ourselves on identical cells and cell-type labels.
Strongest existing link predictor. SCRIPT uses graph attention masked by Hi-C and eQTL priors and is pretrained on 1.3 million cells. It reports a mean AUC of 0.89, against 0.70 for the previous best method, on cis-regulatory relationship prediction across five paired scATAC and scRNA datasets. It is the strongest competitor because its masks inject exactly the external evidence our method lacks.
For fairness we will run it with its released pretraining and evaluate on the same five dataset pairs. Critically, we will also run a held-out-database protocol: Hi-C and eQTL evidence from a subset of tissues is withheld from SCRIPT's mask and used only as test labels. Beating SCRIPT means matching its 0.89 within noise on the standard protocol and exceeding it on the held-out protocol and on links beyond 100 kb.
Distance-window prior. MultiGATE reports 0.703 AUROC for peak-to-gene links against eQTLs on human hippocampus spatial ATAC-RNA data. We will also run our own architecture with a MultiGATE-style distance window substituted for the learned router. This isolates the routing rule from everything else in the model.
Translation baseline. For expression prediction, AIM reports an RMSE of 0.2206 on the 12-donor bone marrow scATAC-to-expression task, against 0.2232 for Cajal. We will match its data splits. We do not need to beat AIM on RMSE. Staying within about 1% is enough, because the contribution is link discovery, and a large accuracy loss would undermine trust in the attention maps.
Ablations of the key ingredient. All ablations share the same backbone, candidate budget $k+m$, training steps, and hyperparameter search budget.
- FastV-style routing: candidates are chosen by attention mass from a shallow full-attention layer.
- Top-$k$ without exploration: the learned router with $m=0$. Plain top-$k$ is known to be a resilient baseline from Which Tokens to Use?.
- Random routing: candidates are drawn uniformly.
- No gate: the full method with the gate removed.
Why not just…?
Why not just correlate peaks with genes across cells? Correlation is cheap, genome-wide, and database-free, so it is a genuine threat on link recovery. Its weakness is that it scores each pair in isolation. It cannot explain away one peak using another, so correlated neighbouring peaks inherit each other's credit, and it produces no expression model. If correlation matches us on distal eQTL recovery in the planted-link and OneK1K checks, the attention machinery is not earning its keep, and we would report that outcome as the finding.
Why not just use a large distance window, since most regulation is cis? This is the most serious alternative for ordinary links. A 1 Mb window probably covers most true cis links. The windowed ablation of our own model answers the question directly. If learned routing adds nothing within 1 Mb and finds no reproducible links beyond it, the honest conclusion is that windows suffice, and the method reduces to a window with better bookkeeping. We treat trans links as exploratory because they are rare and mostly indirect.
Why not just brute-force exact attention with memory-efficient kernels? At roughly $1.5 \times 10^9$ pairs per cell per head, exact attention is costly but not impossible on a cluster. Its result is also the gold standard for fidelity. We therefore train it on a single chromosome, AIM's setting, where exact attention is cheap, and use it as the reference map. If exact training at genome scale proves affordable, the value of our method shrinks to lower compute cost. The fidelity analysis still tells practitioners which cheaper rules are safe for interpretation.
Why not just mask with Hi-C and eQTL databases as SCRIPT does? Masks work where databases exist. By construction, however, they cannot score pairs outside the database, as SCRIPT itself acknowledges. The held-out-database protocol measures exactly this cost.
Proposed Experiments
Training data. We train on the NeurIPS hematopoiesis multiome dataset (63,683 bone marrow cells), which scooby also uses, and on the 12-donor bone marrow data used by AIM.
Link-recovery evaluation. We evaluate against three sources. The first is SCRIPT's five-pair benchmark, under both the standard and held-out-database protocols. The second is cell-type eQTLs from the OneK1K cohort, where a predicted link counts as supported if an eQTL variant for that gene falls in the peak. We stratify this evaluation by distance, reporting links within 100 kb and beyond 100 kb separately, because scooby documents systematic attenuation of distal effects in sequence models. The third is MultiGATE's hippocampus dataset, as a secondary spatial check. The reported metrics are AUROC, AUPRC, and translation RMSE.
First decisive check: fidelity. On a single chromosome, we compare link rankings from each routing rule against the exact-attention model, using Spearman correlation and recall of the exact model's top 1% of links, as a function of the budget $k+m$. The hypothesis fails if learned routing with exploration is not clearly more faithful than FastV-style routing at equal budget.
Second decisive check: planted-link recall. In the semi-synthetic data, we plot recall of planted links against effect strength and distance for each routing rule. We expect attention-mass and window routing to lose weak distal links first, and exploration to recover them at a small cost to strong local ones. If exploration shows no recall advantage for weak distal links, the exploration idea is wrong.
Third decisive check: distal links in real data. We test whether learned routing exceeds the windowed ablation and SCRIPT on real eQTL-supported links beyond 100 kb. If it does not, the practical case for the method rests on fidelity alone.
Wall-clock cost. We report measured wall-clock time, not only theoretical operation counts. FastV notes that dynamic pruning often fails to turn reduced FLOPs into real speedups. Our candidate sets have a fixed size of $k+m$ per gene, which avoids the ragged tensor shapes behind that gap.
References
- AIM: an accurate and explainable model for ATAC to GEX translation and pathway analysis
- SCRIPT: Predicting Single‐Cell Long‐Range Cis‐Regulation Based on Pretrained Graph Attention Networks
- MultiGATE: integrative analysis and regulatory inference in spatial multi-omics data via graph representation learning
- An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models
- AgilePruner: An Empirical Study of Attention and Diversity for Adaptive Visual Token Pruning in Large Vision-Language Models
- An Interpretable Framework for Drug-Target Interaction with Gated Cross Attention
- LookupViT: Compressing Visual Information to a Limited Number of Tokens
- SG-Former: Self-guided Transformer with Evolving Token Reallocation
- SToFM: a Multi-scale Foundation Model for Spatial Transcriptomics
- Cross-Modal Imputation and Uncertainty Estimation for Spatial Transcriptomics
- An interpretable RNA foundation model for exploring functional RNA motifs in plants
- Which Tokens to Use? Investigating Token Reduction in Vision Transformers
- scooby: Modeling multi-modal genomic profiles from DNA sequence at single-cell resolution