Species-Specific Clocks, Shared Lookup: Hybrid SSM-Attention Models for Translational Physiology

Most drugs that succeed in animals fail in humans, and the file gives two distinct reasons. One is structural: rodents lack a macula, their skin and corneas heal differently, and worms have no blood-brain barrier, so anatomy itself breaks the mapping. The other is temporal and less studied: the same physiological program runs at radically different rates and phases in each species — rodent hearts near 600 beats per minute versus 60–80 in humans, inverted circadian metabolic cycles, phosphorylation cascades unfolding in minutes while the gene expression that must be used to predict them is measured hours later. A striking quantitative marker of this second problem is the IMPROVER Species Translation Challenge, where the best methods for predicting rat-to-human signaling from transcriptomics plateaued at roughly 70% balanced accuracy using only static, single-timepoint data (Predicting protein phosphorylation from gene expression).

At the same time, sequence modeling has developed exactly the computational primitive this problem seems to need. Transformers — models that compare every pair of positions — are precise at lookup and variable binding but scale quadratically. State space models (SSMs), such as Mamba, compress a sequence into a fixed-size running state, scaling linearly but losing detail. Theory work has now proved that a particular hybrid ordering — an SSM that first summarizes a long context, feeding an attention module that then performs exact associative lookup — solves variable-binding-under-summarization tasks with up to 6× fewer parameters than pure models, roughly 10% better length generalization, and over 15% better out-of-distribution accuracy (Expressivity-Efficiency Tradeoffs for Hybrid Sequence Models). My claim is that cross-species physiology translation has precisely this shape: each species' time course is a long, rate-distorted context that must be compressed at that species' clock rate, then queried exactly. No existing cross-species model does either half of this — all are static snapshots built from dense attention or shallow autoencoders.

Core Claim

The translational gap, in its temporal component, is a time-warp plus variable-binding problem: homologous processes carry species-specific rates and species-specific molecular markers, and a model must (a) absorb the rate difference while compressing each species' history, and (b) still bind the right markers exactly when making a prediction in the other species. I propose a species-conditioned hybrid encoder–decoder — call it the Time-Warped Hybrid — in which each species gets its own SSM encoder whose internal timescale is modulated by a learned, species-identifying input, while a single shared attention decoder performs the cross-species lookup. The design deliberately instantiates the SSM→Attention ordering proved optimal by Cooper et al., and the reversal (attention first, SSM second) is known to give no asymptotic advantage, which is why the ordering is fixed rather than searched.

Concretely, a Mamba-style layer updates its state with an input-dependent step size $\Delta$ that controls how fast the running summary forgets and integrates. I make this step size species-conditioned:

$$\Delta_s(t) = \exp\!\big(g(\phi_s)\big)\cdot \Delta_0(t),$$

where $\phi_s$ is a learned species embedding (one per species), $g$ is a small linear map, and $\Delta_0(t)$ is the base input-dependent step size. Initializing $g$ so that the ratio between rodent and human encoders matches gross physiological ratios (an ~8.6× heart-rate factor, a 12-hour circadian phase inversion, a minutes-versus-hours signaling lag) gives the model a strong inductive bias that fine-tuning can refine. The encoders' outputs — fixed-size latent summaries of each species' trajectory — are concatenated and passed to a shared attention decoder that predicts target-species quantities. This also gives a principled resolution to a documented contradiction in the literature: GeneCompass assumes regulatory logic is conserved across shared orthologs and gains from pooling human and mouse data, while SATURN shows homologous genes can drift functionally (the Myoc gene clusters with different macrogene groups in mice, pigs, and humans). The Time-Warped Hybrid lets the dynamics be species-specific (per-species encoders) while the decision logic is shared (the attention decoder), accommodating both conservation and drift rather than committing to one.

Research Program

Three things get built or studied. First, the architecture itself: per-species selective SSM encoders with the conditioned step size above, a shared attention decoder, and a macrogene-style input vocabulary following SATURN — genes embedded by protein language models and clustered, so non-orthologous and duplicated genes are retained rather than discarded as in the 17,465-ortholog vocabulary of GeneCompass. Second, a pretraining corpus of simulated multi-species trajectories: coupled dynamical systems (oscillators with pharmacokinetic-style decay, switching cascades) run under species-specific warps and marker relabelings, so the model first learns the skill of rate-invariant summarization before touching scarce biological data. Third, an analysis program: layer-wise attribution of where species-specificity lives (encoder timescales versus decoder attention patterns), and an attention-allocation study asking how much attention the cross-species binding actually needs — directly extending the finding from Some Attention is All You Need for Retrieval that retrieval collapses to 0% without attention yet survives with as few as 15% of attention heads.

Proposed Experiments

The first experiment is a synthetic benchmark I would construct: species-warped S2FC. Following the Structured Two-Layer Function Composition tasks of Cooper et al. — extract a control variable from context, then execute a variable-bound lookup — I generate contexts in which the "donor species" stream is time-warped by factors sampled around known physiological ratios and uses a relabeled token vocabulary, and the model must answer queries in the "target species" vocabulary. Metrics are accuracy versus warp magnitude, parameter count at fixed accuracy, and length generalization (train on short sequences, test long), compared against a pure Transformer, a pure SSM, an unconditioned hybrid, and a reversed attention→SSM hybrid, all at matched parameter counts and training budgets. The hypothesis, grounded in the cited theory, is that only the species-conditioned SSM→attention ordering holds accuracy as warp grows.

The second experiment is the real-data test: the IMPROVER Species Translation sub-challenge, predicting 16 phosphoprotein states from 13,841-gene expression across 52 rat stimuli split 26 train / 26 test, with expression at 6 hours and phosphorylation at 5 and 25 minutes (Predicting protein phosphorylation from gene expression). The minutes-to-hours mismatch is not a nuisance here; it is the time-warp the model must internalize. The hybrid is pretrained on the simulated corpus plus cross-species single-cell atlases (scCompass-126M scale), then fine-tuned on the 26 training stimuli with the same cross-validation budget as every baseline. Success means beating the ~70% balanced accuracy and AUPR ~0.4 plateau on held-out stimuli. I flag honestly that 26 training stimuli is a small target for a deep model; this is exactly why pretraining carries the load and why the synthetic benchmark carries the mechanistic claim.

The third experiment is the allocation and ablation study: needle-in-a-haystack retrieval where the needle is defined in the donor species' vocabulary, sweeping attention head fraction in the decoder. Based on Michalak and Abreu, who found 0% accuracy with attention removed and near-perfect accuracy with 15% of heads (27 of 32 in Jamba), I expect a sharp threshold; the open scientific question is whether cross-species binding is more attention-hungry than same-vocabulary retrieval, as dnaHNet leaves the attention requirement of biological grammar unresolved. A fourth, qualitative check applies the model to circadian liver-organoid chronotoxicity time courses, where rodent and human rhythms are inverted (Chronotoxici-Plate) and where islet clock phase divergence is documented at the cellular level (Pancreatic α- and β-cellular clocks); no reported result found on Lacuna for a predictive benchmark there, so this is a sanity check on whether the learned $g(\phi_s)$ recovers the 12-hour phase inversion without being told.

Baselines

The naive reference point is direct value transfer: assume the donor species' measured quantity is the human prediction, with no learned model. This is the implicit baseline that preclinical practice encodes, and its failure mode is documented systemically rather than numerically — the meta-analysis of animal models of cerebral ischemia (Treatment of acute cerebral ischemia using animal models) reports that the majority of animal-effective compounds fail in human trials; no reported result found on Lacuna for direct transfer accuracy on the IMPROVER split. The strongest existing method on this exact question is the IMPROVER winner family: Team 49's pipeline of moderated t-test gene selection (an average of only 3.4 genes per target protein) plus Linear Discriminant Analysis, and Team 75's ridge regression / RBF support vector regression, which reached ~70% balanced accuracy, AUPR ~0.4, and Pearson correlation ~0.7 on the blinded 26-stimulus test set (Predicting protein phosphorylation from gene expression). I will reimplement Team 49 and Team 75 and tune them with the same cross-validation budget on the same 26 training stimuli given to the hybrid, and beating the baseline means exceeding ~70% balanced accuracy and ~0.4 AUPR on the identical held-out split with a fixed, pre-registered prediction rule. Two deep competitors get the same data and compute: a GeneCompass-style dense Transformer fine-tuned on the same task (that model reports a 15.4% MSE reduction over Geneformer in perturbation prediction and 3–8% macro-F1 gains in cell typing, so it is the proven deep baseline in this domain, though on different tasks), and a SATURN-style static macrogene autoencoder (85.8% median label-transfer accuracy on frog→zebrafish, its strongest reported number, on a different benchmark). The key ablation is the hybrid with its species conditioning removed — identical architecture, $\Delta$ not modulated by $\phi_s$ — which isolates whether the claimed ingredient, the learned clock warp, is what buys the gain rather than generic hybrid capacity; a second ablation removes decoder attention entirely, where the 0% retrieval collapse reported by Michalak and Abreu predicts failure.

Why not just…?

Why not just use a pure Transformer, since GeneCompass and BrainBeacon already work at 126–133 million cells? These models prove dense attention works for static cross-species snapshots; they do not address rate-divergent time courses at all, and BrainBeacon explicitly models no temporal or pharmacokinetic mapping. Where long contexts do appear, the cost is real: on an H100, standard Transformers hit out-of-memory errors at $2^{19}$ nucleotides while the hybrid backbone of dnaHNet used 3.89× fewer FLOPs and 3.5× less memory than StripedHyena2 with over 3× inference speedup. I am honest that at IMPROVER's scale (13,841 genes, short sequences) a Transformer is perfectly feasible — which is why the pure-Transformer baseline is included at matched budget, and why the architectural claim rests on the warped synthetic benchmark and scaling regime, not on IMPROVER alone.

Why not just use a pure SSM, which handles continuous time natively and scales linearly? Pure SSMs fail exactly at the binding step: ablating attention collapses retrieval to 0% accuracy in Michalak and Abreu, even with Just-Read-Twice prompting, and Cooper et al. prove pure SSMs need parameter counts scaling linearly with the hidden dimension on these tasks. Cross-species prediction requires precise binding of species-specific markers, so a pure SSM is ruled out by prior evidence, not by preference. The proposal keeps the SSM only where it is provably strong — compressing rate-warped history — and the no-attention ablation tests this directly.

Why not just use a few sentinel genes and a linear classifier, since Team 49 tied complex models with 3.4 genes per protein? This is the most serious threat, because on the IMPROVER data itself the shallow approach is the incumbent and the data are too small to reward depth by default. The evidence for a ceiling is that all shallow winners — LDA, PCA voting, ridge, RBF-SVR — plateaued at ~70% balanced accuracy and AUPR ~0.4 (IMPROVER) at a single static timepoint, while foundation-scale models show large gains on related transfer tasks (GeneCompass's up-to-36% mouse annotation improvement). But the honest position is: if the pretrained hybrid, tuned with the same budget, does not beat 70% on the 26 held-out stimuli, the temporal hypothesis is not supported at this data scale, and the proposal's claim would retreat to the synthetic benchmark and scaling analysis. That decisive comparison is built in as the primary endpoint.

Why not just reuse SATURN's ESM2 macrogene embeddings and metric learning? SATURN solves vocabulary divergence elegantly — it is why I adopt macrogenes as the input vocabulary — but its conditional autoencoder is static: it maps one gene-count vector to one embedding, with no mechanism for time courses, rate warps, or the minutes-to-hours mismatch that defines the IMPROVER task, and no ability to learn regulatory sequence grammar the way dnaHNet learns codon periodicity from raw sequence. The SATURN-style autoencoder is included as a baseline under identical conditions so the comparison is empirical rather than rhetorical.

Decisive Checks

The proposal succeeds if three results hold: species-conditioned hybrids uniquely survive large warps on the synthetic benchmark where unconditioned and pure models degrade; the IMPROVER held-out balanced accuracy exceeds the ~70% shallow-model plateau under matched tuning; and the attention-allocation sweep shows cross-species binding needs more attention heads than same-vocabulary retrieval, quantifying for the first time how much exact lookup temporal translation demands. It fails informatively if the hybrid matches but does not beat the sentinels on IMPROVER — which would localize the translational gap's remaining difficulty in data scarcity rather than model class, itself a publishable answer to the question the IMPROVER ceiling left open.