A protein is synthesized as a linear chain of amino acids, but it functions as a complex, dynamic three-dimensional machine. In recent years, Protein Language Models (pLMs) have revolutionized computational biology by treating these amino acid sequences like natural language text. By training on massive databases of primary sequences, pLMs successfully capture the "grammar" of evolution.
However, this text-like abstraction introduces a fundamental blind spot: the omission of spatial context. In a one-dimensional sequence, two amino acids might be separated by hundreds of positions, yet when the protein folds into its native state, those same residues may sit directly adjacent to one another in 3D space. Together, they form critical structural motifs, catalytic active sites, or drug-binding pockets. As Xu et al. observe, relying exclusively on sequence-based representations creates a severe gap in understanding how physical shape influences biological function.
To build models capable of true physical reasoning, researchers are developing architectures and pretraining strategies that explicitly bridge this representation gap by integrating 3D geometric coordinates into sequence-based models.
Where Pure Sequence Models Fail
The limitations of sequence-only models become glaringly obvious in tasks that depend on precise physical orientation and steric constraints.
Binding Affinity and Molecular Docking
Predicting how a small molecule drug or a nucleic acid binds to a target protein requires an exact understanding of the binding pocket's geometry. Models that rely solely on 1D sequences ignore the 3D structural folding and pocket topology that dictate physical binding (Bal et al.; Wang et al.). Without explicit 3D structural coordinates, models struggle with tasks highly sensitive to geometric orientations, such as estimating binding affinities, because they cannot simulate how molecules physically fit together in a 3D environment (Kalifa et al.).
Interface and Antibody Design
In generative tasks, such as designing antibodies to neutralize specific pathogens, sequence-only models often lack the spatial intuition required to engineer functional protein-protein interfaces. Without incorporating 3D structural information, generative frameworks struggle to account for the complex geometric interactions and physical fit required for high-affinity binding (Ren et al.).
Why Sequence Pretraining Doesn't Inherently Learn Space
A common assumption is that given enough data and parameters, a transformer trained on sequences will implicitly learn the underlying 3D physics of protein folding. However, standard self-supervised objectives—such as masked language modeling—lack the explicit 3D inductive biases required to fully model spatial relationships (Rao et al.).
While a massive language model can achieve high sequence-to-text fluency, Chen et al. demonstrate that this textual fluency does not inherently translate to high-fidelity structural modeling. Similarly, proxy pretraining tasks like sequence reordering can capture long-range dependencies, but because they lack explicit access to 3D atomic coordinates, they remain limited in their ability to precisely map the physical docking constraints of molecular pockets (Zhang et al.).
Strategies for Integrating 3D Geometry
To overcome these limitations, the field has converged on several distinct mechanisms to inject spatial awareness into protein representations.
Strategies for Spatial Integration:
├── 1. Direct Geometric Pretraining ──> Learn representations entirely from 3D coordinates (e.g., Graphs/Point Clouds)
├── 2. Cross-Modal Fusion ──> Combine 1D sequence tokens with 3D structural features
├── 3. Post-Training Alignment ──> Align frozen pLM embeddings with GNN structural features
└── 4. Structural Inductive Biases ──> Constrain sequence models using 2D contact/distance maps
1. Direct Geometric Pretraining
Instead of treating the protein as a string of characters, some architectures bypass sequences entirely during pretraining to operate directly on 3D structures. For instance, Zhang et al. developed GearNet, a geometry-aware relational graph neural network. By pretraining on 3D coordinates using multiview contrastive learning and geometric self-prediction tasks, the model explicitly captures structural motifs formed by residues that are distant in the sequence but physically adjacent in space.
2. Cross-Modal Fusion
Rather than choosing between sequence and structure, cross-modal architectures fuse both representations to get the best of both worlds. Sequence models provide deep evolutionary context, while structural models provide physical constraints. Jing et al. implemented this in CrossBind, fusing 3D atom-level structural data (via point clouds) with 1D sequence data (via pLMs) to identify nucleic-acid-binding residues, significantly outperforming single-modality approaches.
3. Post-Training Representation Alignment
Training massive multimodal models from scratch is computationally expensive. An alternative strategy is to take a powerful, frozen sequence model and project its representations into a structurally aware space. Chen et al. introduced a lightweight post-training method that aligns sequence representations from pLMs with structural features derived from Graph Neural Networks (GNNs). This alignment injects deep structural awareness into the model, allowing it to retain the speed of a sequence-only model at inference time while benefiting from geometric insights.
4. Structural Inductive Biases
When explicit 3D coordinates are missing or too noisy, 2D abstractions of structure—such as contact maps or pairwise residue distance matrices—can serve as an intermediate bridge. By incorporating contact maps as an explicit inductive bias, models can constrain their attention mechanisms to respect physical proximity boundaries, preventing the model from predicting interactions that are physically impossible in a 3D environment (Bal et al.).