Models that predict whether a small-molecule drug will bind a protein must routinely answer for proteins they have never seen during training — a "cold" target with no measured interactions. The best current cold-start models do this with fine-grained cross-modal attention: each atom of the drug inspects individual residues of the protein, so the model can reason about local binding contacts rather than compare two pooled vectors. But these models inherit a hard ceiling from their protein encoders. Standard protein language models are Transformers, whose self-attention cost grows with the square of sequence length, so proteins get cropped to roughly the first thousand residues — NetGO 3.0 crops at 1000 residues, and FusOn-pLM caps at 2,000 out of memory necessity. As Cross-protein transfer learning substantially improves disease variant prediction points out, fixed input windows leave nearly half of the human proteome — including many long, multi-domain kinases and receptors that are prime drug targets — effectively inaccessible.
Linear-scaling state-space models (SSMs) — recurrent networks whose memory and compute grow only proportionally to sequence length — have already removed this ceiling for proteins: PRIMA processes concatenated protein complexes of 16,384 residues on a single GPU, running 11.5× faster and using 115× less memory than an equal-size Transformer at that length. Yet PRIMA fuses molecules only by writing them into one long token stream; it has no mechanism for a compact molecular graph to interrogate a long protein chain, and it has never been evaluated for cold-start small-molecule screening. This proposal closes that gap: a drug–target affinity model whose protein side is a full-length bidirectional SSM and whose fusion layer is gated cross-attention from drug atoms to protein residues, evaluated under strict cold-start splits with special attention to targets longer than the truncation ceiling.
Core Claim
The claim is twofold. First, the two known bottlenecks — quadratic protein encoding and the $O(L_1 \times L_2)$ cross-attention map — can be removed simultaneously and independently: an SSM protein encoder makes the protein side linear in length, and because a drug has only $m \approx 20$–$50$ atoms while the protein has $L$ residues, atom-to-residue cross-attention costs $O(m \cdot L)$, which is already linear in the long dimension. No truncation is required anywhere. Second, this length asymmetry makes modality dominance worse, not better: a tiny drug graph querying thousands of residues is exactly the regime where An Interpretable Framework for Drug-Target Interaction with Gated Cross Attention showed that un-gated cross-attention collapses the drug representation (its cosine-similarity diagnostic), so gating must be treated as a first-class design constraint rather than a refinement.
Concretely, the model encodes the protein with a bidirectional Mamba stack (following PRIMA's recipe, pretrained by masked-language modeling on UniRef50 and post-trained on STRING protein–protein interaction pairs), encodes the drug as a small molecular graph with a graph neural network, and fuses them with gated, sparsified cross-attention:
$$\text{CA} = \mathrm{Sparsemax}\!\left(\frac{Q_{\text{atom}} K_{\text{res}}^{\top}}{\sqrt{d}}\right) V_{\text{res}}, \qquad H_{\text{atom}}' = g \odot \text{CA} + (1 - g) \odot H_{\text{atom}}$$
where $Q_{\text{atom}}$ are query vectors from the $m$ drug-atom embeddings, $K_{\text{res}}, V_{\text{res}}$ come from the $L$ residue embeddings, $d$ is the head dimension, and $g$ is a learnable gate that interpolates between the cross-modal context and the original atom features, as introduced by the GCA paper. Sparsemax replaces softmax so attention concentrates on plausible binding residues instead of diffusing across thousands of positions. A reverse pass (residues querying atoms) has the same $O(m \cdot L)$ cost and gives each residue a drug-context summary.
Research Program
Three components, in order. The first is architectural: build the encoder described above, reusing PRIMA's BiMamba-S design and its pretraining recipe (5 million UniRef50 sequences for masked-language modeling, then STRING post-training), a graph-isomorphism network for drug atoms following HeteroBindNet, and the gated Sparsemax fusion of GCA. The drug encoder additionally gets interaction-network pretraining on STITCH chemical–chemical interactions, adopting the transfer recipe of C2P2, which showed that pretraining on interaction graphs — not just single molecules — supplies the priors that make cold generalization possible. Crucially, C2P2 also reported that pairing such independently pretrained encoders can cause destructive latent-space misalignment under naive concatenation fusion; our cross-attention fusion is the proposed antidote, and we will test that directly by tracking alignment diagnostics during fine-tuning.
The second component is a diagnostic evaluation protocol that the field currently lacks: length-stratified cold splits. Existing cold-start evaluations pool all targets together, so a model can post good numbers while silently failing on exactly the proteins that motivated the architecture. We will partition each cold-target test set into buckets — at most 1,022 residues (the ESM-2 window), 1,022 to 2,048, and above 2,048 — and report metrics per bucket. For the over-2,048 bucket, truncating baselines see only a crop, so the comparison isolates the value of full-length context.
The third component is a scientific question borrowed from the tension between two cited papers: ColdDTI found that quaternary (multi-chain) structural features give diminishing returns in cold-start prediction, while PRIMA shows full-length multi-chain context matters for interaction modeling. We will re-run ColdDTI's four-level structural ablation on top of our full-length encoder to determine whether the quaternary level's failure was an artifact of Transformer truncation or a genuine redundancy — either answer is informative, and no cited work can currently distinguish the two.
Baselines
The naive reference point is pooled late fusion: encode the protein with a pretrained protein language model, average all residue embeddings into one vector, encode the drug with a graph network, concatenate, and predict with a small MLP. This is the de facto default in the literature — it is what NetGO 3.0 does to compress cropped sequences, and what C2P2 uses for fusion — and the cited evidence says it leaves real performance on the table: ProtoCol showed that a 35M-parameter model with token-level scoring beats a pooled 650M-parameter ESM-2 by nearly 7 points on remote-homolog retrieval, and GCA showed pooled/un-gated fusion degrades Davis MSE from 0.393 to 0.438 relative to its gated variant. Beating this baseline is necessary but not sufficient; it mainly checks that our pipeline is sound.
The strongest existing competitor is ColdDTI, which holds the best reported cold-start numbers we found on Lacuna: an AUC of 0.751 on the BindingDB cold-protein split (where most sequence-only baselines sit below 0.70) and 0.818 on the Human cold-pair split. It cross-attends multi-granularity drug tokens against four hierarchical levels of protein structure, but its ProtTrans backbone is quadratic and truncates long chains. We will rerun ColdDTI on our identical splits with its released protocol, matched tuning budget (same search space, same early stopping on the same validation folds), and report per-length-bucket AUC alongside its pooled number. Beating it means exceeding 0.751 pooled AUC on BindingDB cold-protein, and — the sharper test — a materially larger margin on the over-2,048-residue bucket, where its encoder is structurally handicapped. On the regression side, HeteroBindNet is the strongest cold-split method on KIBA and Davis; it reports 14.95% MSE and 5.05% CI improvement over the DMFF-DTA baseline on the KIBA cold-drug split, but no absolute value for that split is reported on Lacuna, so we will reproduce it under our protocol and treat the reproduced number as the target. GCA provides a third anchor with an absolute figure — Davis MSE 0.393 on a random split — though it was never evaluated on cold splits, which we will also do for it.
Three ablations isolate the proposal's ingredients. Swapping the SSM protein encoder for a truncated ESM-2 encoder while keeping the gated fusion isolates the long-context ingredient. Removing the gate (plain softmax cross-attention) isolates the gating ingredient, evaluated with GCA's cosine-similarity identity-loss diagnostic. Replacing cross-attention with PRIMA-style concatenation — tokenizing the drug's SMILES string and appending it to the protein sequence in one SSM stream — isolates the value of heterogeneous cross-modal fusion. All ablations share the same pretraining data, parameter count, and tuning budget as the full model.
Why not just…?
Why not just truncate at 1,022 or 2,000 residues, since binding pockets are local? This is the most serious objection, and the honest answer is that for short kinase-heavy benchmarks like Davis it may well suffice — indeed we expect truncated baselines to be competitive on the under-1,022 bucket, and if they match us there, that is the correct finding. But the cited evidence says locality is not the whole story: distal signals governing localization and regulation sit in the center and far termini of long chains and are permanently discarded by cropping (documented in the truncation analysis around NetGO 3.0), Cross-protein transfer learning reports that fixed windows lock out nearly half the human proteome, and PRIMA achieves roughly 20% lower perplexity than a Transformer baseline on sequences longer than 5,000 residues. Our length-stratified cold splits are designed precisely to settle this: if the gains concentrate in the over-2,048 bucket, truncation is insufficient; if they do not, we will report that the long-context machinery was not the binding constraint.
Why not just concatenate the drug and protein into one state-space stream, PRIMA-style? It is cheap, and we will run it as an ablation rather than argue it away. But PRIMA itself is a 7.3M-parameter proof of concept whose authors note it has not resolved whether the network learns true inter-chain signals or leans on intra-chain statistics, and a single token stream over a SMILES string concatenated with residues gives every position the same representational form and no explicit atom-to-residue contact structure — the very structure that let HeteroBindNet recover about 92% of real atomic contacts and GCA localize binding residues at nearly 90% accuracy with one attention head. If the concatenation ablation matches our cross-attention model, that would be a genuinely useful negative result about fusion design, and we would report it as such.
Why not just stretch a Transformer instead — local attention plus 4-bit quantization? Local attention with 4-bit quantization (Scaling Up ESM2 Architectures for Long Protein Sequences Analysis: Long and Quantized Approaches) doubles the ESM-2 context to 2,048 residues, which covers our middle bucket, and we will include it as a baseline at that limit. But the underlying cost still grows quadratically, so the over-2,048 bucket and multi-chain complexes remain out of reach, and the approach does not address the cross-attention side at all. If the quantized Transformer matches us at 2,048 and we only win beyond it, the proposal's value is then specifically the full-proteome regime — which is the regime that motivated it.
Proposed Experiments and Decisive Checks
Evaluation proceeds on three fronts. Cold-start classification uses the BindingDB, BioSNAP, Human, and DrugBank cold splits from ColdDTI, reporting AUC per cold regime (cold drug, cold protein, cold pair) and per length bucket; the headline targets are 0.751 AUC on BindingDB cold-protein and 0.818 on Human cold-pair. Affinity regression uses the KIBA and Davis cold splits from HeteroBindNet, reporting MSE and concordance index, with GCA's Davis MSE of 0.393 and HeteroBindNet's overall Davis MSE of 0.176 as reference points (no absolute KIBA cold-split number is reported on Lacuna for HeteroBindNet, so we reproduce it). Where existing benchmarks contain too few proteins above 2,048 residues, we will curate a supplementary cold-target set from BindingDB and PDBBind entries filtered by length, following the same split-construction rules, and disclose its construction.
Three checks decide the scientific claims, beyond aggregate metrics. First, a scaling curve: memory and runtime as a function of protein length, up to 16,384 residues, for our model versus the ESM-2-based cross-attention baseline, demonstrating the linear regime that PRIMA established for single streams now holds for cross-modal fusion. Second, fusion health: GCA's cosine-similarity identity-loss diagnostic run at each length bucket, testing whether the gate still prevents drug-representation collapse when the residue side grows from hundreds to thousands of positions — no cited work has measured gating behavior in this regime. Third, the ColdDTI ablation replay: primary, secondary, tertiary, and quaternary structural features added on top of our full-length encoder, answering whether quaternary features were redundant because of truncation artifacts or genuinely low-value for cold generalization. As a practical resource, the released ProtoCol codebase provides the CUDA-accelerated MaxSim late-interaction implementation we will reuse for a token-level late-interaction variant — a fourth ablation that keeps residue-level granularity without attention, probing whether the gating and alignment machinery matters or whether granularity alone carries the gains.
References
- NetGO 3.0: Protein Language Model Improves Large-scale Functional Annotations
- FusOn-pLM: a fusion oncoprotein-specific language model via adjusted rate masking
- Cross-protein transfer learning substantially improves disease variant prediction
- PRIMA: a bidirectional state-space architecture and training approach for sequence modelling of protein-protein interactions
- An Interpretable Framework for Drug-Target Interaction with Gated Cross Attention
- Graph-Based Cross-Modal Learning for Drug–Target Affinity Prediction
- Mitigating cold-start problems in drug-target affinity prediction with interaction knowledge transferring
- Attending on Multilevel Structure of Proteins enables Accurate Prediction of Cold-Start Drug-Target Interactions
- ProtoCol: Late Interaction Retrieval for Protein Homolog Search
- Scaling Up ESM2 Architectures for Long Protein Sequences Analysis: Long and Quantized Approaches