Distilling Structural Knowledge into Protein Sequence Models

This direction is merged into another direction: structural data dependency (structural data scarcity). Its last version stays readable and citable here.

Version 1 of 1133 papers141 conceptsfrozen 2026-10-04

The central challenge in computational biology is a massive data asymmetry. We have access to hundreds of millions of protein sequences, but only a tiny fraction of these have experimentally determined 3D structures. While 3D coordinates are the "ground truth" for understanding how a drug binds to a target or how a mutation causes disease, obtaining them via X-ray crystallography or NMR is a slow, expensive bottleneck.

A new research direction seeks to resolve this by building models that are structure-aware during training but structure-free during inference. By distilling high-resolution structural knowledge into sequence-based architectures, researchers are creating models that "imagine" the 3D physics of a protein while looking only at its 1D amino acid string.

The Bottleneck of Explicit Coordinates

Traditional structure-based models, such as AtomNet or Struct2Graph, require explicit 3D atomic coordinates as input. This creates a "structure gap": if you don't have a crystal structure or a high-confidence AlphaFold prediction, these models cannot function. Even when predictions are available, generating them for millions of compounds in a virtual screen adds massive computational overhead.

As Daniel Peñaherrera and David Ryan Koes argue in their work on SI-PLM, the goal is to bridge this gap. They demonstrate that by "infusing" structural knowledge into a transformer during its training phase, the model learns to extract structural features directly from the sequence. At test time, the 3D data is no longer needed, but the model’s internal representations still reflect the spatial reality of the protein.

Implicit Structure and Imagination

How does a model "learn" structure without seeing it at inference? One prominent mechanism is structural distillation.

In the Implicit Structure Model (ISM), Jeffrey Ouyang-Zhang and colleagues use an atomic autoencoder to teach a standard sequence model (like ESM2) to "imagine" 3D coordinates. By forcing the sequence model to predict the latent representations of an encoder that did see the 3D structure, the sequence model learns to prioritize the residues that are spatially close, even if they are far apart in the 1D chain.

Another approach, pioneered by Liang Hong in the ProSST model, involves "digitizing" structure. By quantizing 3D shapes into discrete tokens, a Transformer can process sequence and structure as a unified language. This allows the model to learn the "grammar" of protein folds, which it can then apply to sequence-only tasks with much higher accuracy than models trained on text alone.

Structure-Free Interaction Prediction

This shift is particularly transformative for drug discovery and molecular recognition. Historically, predicting how a ligand binds to a protein required docking simulations—physics-heavy calculations that rely on precise 3D "pockets."

Recent work has moved toward "structure-free" interaction modeling:

  • LINKER, developed by Truong Son Hy, predicts specific biochemical interactions (like hydrogen bonds) using only sequences and SMILES strings. It achieves this by training on 3D co-crystal data but designing the architecture to operate on 1D inputs at scale.
  • PepMLM, introduced by Tianlai Chen and Madeleine Dumas, uses a span-masking strategy to design therapeutic peptide binders. By bypassing the need for explicit target structures, it can design binders for "undruggable" proteins that lack stable 3D folds.

The intuition behind this direction is that the primary sequence of a protein already contains the "blueprint" for its fold. As Peyton Greenside and Anshul Kundaje showed, molecular recognition can be decoded by pairing sequence motifs with chemical substructures.

By using 3D data as a "teacher" during training, we are essentially teaching the model to look for the evolutionary and chemical signals in the sequence that correspond to spatial proximity. Once the model learns these signals, it no longer needs the 3D "crutch." This allows for a 10,000-fold speedup in tasks like antibody affinity maturation, as demonstrated by Kevin Jin and Lurong Pan, enabling rapid responses to emerging viral threats where 3D structures are not yet available.

Beyond Proteins: Materials and RNA

This philosophy of "distilling out" the dependency on 3D coordinates is spreading to other domains. In materials science, Linda Hung and Weike Ye developed XxaCT-NN, which predicts material properties from elemental composition and X-ray patterns rather than precise atomic coordinates. Similarly, in RNA biology, models like PrismNet by Kui Xu integrate in vivo structural data to predict interactions, moving the field toward a more dynamic, "shape-aware" understanding of molecular biology that doesn't stop at the sequence.

Papers in this version

Showing 60 of 133 papers.

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_distilling_structural_knowledge_into_pro_61846c0d,
  title = {Distilling Structural Knowledge into Protein Sequence Models},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/distilling-structural-knowledge-into-protein-sequence-models/rdr_61846c0d382d44b89dd6bdd1d539b277},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_distilling_structural_knowledge_into_pro_61846c0d_v1,
  title = {Distilling Structural Knowledge into Protein Sequence Models},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/distilling-structural-knowledge-into-protein-sequence-models/rdr_61846c0d382d44b89dd6bdd1d539b277/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}