Deep learning has historically treated the discrete world of natural language and the continuous world of physical geometry as fundamentally separate modalities. Processing text relies on large vocabularies of categorical tokens optimized for Transformers, while processing 3D structures—such as proteins, small molecules, and computer-aided design (CAD) models—traditionally requires specialized architectures like Graph Neural Networks (GNNs) or diffusion models capable of handling continuous $(x, y, z)$ coordinates.
While continuous geometric models are highly expressive, they suffer from significant computational overhead, struggle to scale as efficiently as large language models, and are highly sensitive to coordinate noise.
The core insight of structural tokenization is that continuous 3D geometric data can be quantized into a discrete, one-dimensional vocabulary of "tokens" without losing critical spatial information. By translating shapes into sentences, researchers can unlock the scalability, speed, and autoregressive capabilities of standard Transformer architectures for complex geometric reasoning.
The Mechanism: From Coordinates to Codebooks
To represent a continuous 3D structure as a discrete sequence, models must map spatial relationships to a finite set of representative geometric patterns. This is typically achieved by analyzing local geometric environments and applying vector quantization.
Continuous 3D Coordinates
│
▼
[Local Geometric Descriptors] (e.g., Dihedral Angles, Distance Matrices)
│
▼
[Vector Quantization / Codebook Lookup]
│
▼
Discrete Structural Tokens (e.g., "Token #42", "Token #107")
In structural biology, for example, instead of tracking raw spatial coordinates, models look at invariant local features, such as the folding angles of a protein backbone or the spatial arrangement of neighboring residues. As demonstrated by Cheng Tan et al. in MeToken: Uniform Micro-environment Token Boosts Post-Translational Modification Prediction and Lirong Wu et al. in MAPE-PPI: Towards Effective and Efficient Protein-Protein Interaction Prediction via Microenvironment-Aware Protein Embedding, these continuous local features are compressed into a discrete vocabulary of "micro-environments" using a learned codebook.
Once this codebook is established, any continuous 3D structure can be serialized into a 1D sequence of categorical tokens. In Learning the Language of Protein Structure, Jérémie DONA et al. show that a vector-quantized autoencoder can compress 3D protein backbones into discrete tokens. This allows standard, off-the-shelf generative models (like GPT) to be trained directly on geometric sequences, generating physically viable, novel 3D structures just like generating text.
Why It Works: The Intuition Behind Discretization
Discarding continuous precision in favor of discrete tokens might seem counterintuitive, but it works remarkably well due to three core factors:
1. Bridging the Sequence-Structure Modality Gap
In many biological and physical systems, geometry does not exist in a vacuum; it is intimately tied to an underlying discrete sequence. In proteins, the linear amino acid sequence dictates the final 3D fold. By expanding a language model's alphabet to include both sequence and structural tokens, architectures can process sequence and shape simultaneously.
For instance, Jin Su et al. developed SaProt: Protein Language Modeling with Structure-aware Vocabulary, which embeds both the amino acid type and its local geometric token into a single unified input stream. This design choice allows standard sequence-based transformers to leverage 3D spatial relationships without the computational overhead of graph neural networks.
2. Robust Regularization and Denoising
Continuous geometric data is inherently noisy. Experimental structures obtained via crystallography often contain minor distortions, and predicted structures can have geometric artifacts.
The authors of ProSST: Protein Language Modeling with Quantized Structure and Disentangled Attention note that converting continuous coordinates into discrete tokens serves as a robust regularization technique. Because vector quantization maps a range of similar continuous configurations to the same discrete token, it filters out high-frequency coordinate noise. This effectively bridges the domain gap between pristine simulated structures used during training and noisier experimental structures encountered in the real world.
3. Extreme Data Compression
3D coordinate files are bulky and computationally expensive to process at scale. By mapping recurring spatial motifs to a compact symbolic alphabet, structural tokenization compresses data representations dramatically.
Zhangyang Gao et al. demonstrate in FoldToken: Learning Protein Language via Vector Quantization and Beyond that tokenization can compress gigabytes of coordinate data into megabytes of discrete sequences. This compression makes it computationally feasible to train massive generative models on vast structural databases that were previously too expensive to process in continuous formats.
Generalizing Beyond Biology
The utility of structural tokenization extends far beyond macromolecular structures to any domain where continuous geometry can be broken down into recurring structural primitives.
Small Molecules
In Geometry Informed Tokenization of Molecules for Language Model Generation, Xiner Li et al. introduce a method that transforms 3D molecular geometries into 1D discrete sequences using spherical coordinates and canonical atom ordering. This approach ensures geometric invariance while allowing standard autoregressive language models to generate 3D molecules up to 100x faster than traditional diffusion models.
Computer-Aided Design (CAD)
CAD models are traditionally stored as complex Boundary Representations (B-reps) containing precise mathematical surfaces. Yongkang Dai et al. show in BRepFormer: Transformer-Based B-rep Geometric Feature Recognition that treating individual geometric faces as discrete tokens allows standard Transformer architectures to operate directly on precise mathematical surfaces, avoiding the information loss associated with voxels or point clouds.
Unlocking Sub-Linear Search Speeds
Perhaps the most dramatic practical validation of structural tokenization is in database retrieval. Traditional 3D structural alignment requires computationally expensive geometric superposition algorithms to find similar shapes.
By discretizing 3D tertiary interactions into a 1D "structural alphabet," Michel van Kempen et al. developed Fast and accurate protein structure search with Foldseek. Because the 3D structure is reduced to a string of characters, the complex geometric optimization problem is transformed into a text-like sequence alignment task. This allows the search tool to leverage highly optimized text-retrieval algorithms—such as k-mer prefiltering and SIMD-vectorized local alignment—speeding up database queries by four to five orders of magnitude while maintaining the sensitivity of traditional 3D alignment methods.
Similarly, in a purely geometric context, Ronald Ayoub demonstrates in RUPEE: A fast and accurate purely geometric protein structure search that decomposing geometric sequences into overlapping n-gram shingles allows web-scale text retrieval techniques, like Locality Sensitive Hashing (LSH), to execute rapid, non-superpositional structural comparisons.
Key Foundational Work
Small Libraries of Protein Fragments Model Native Protein Structures Accurately (2002) — contributed 1 concept elements to this direction.
Rule-based approaches to comparative modelling (1993) — contributed 1 concept elements to this direction.