Proteins are three-dimensional shapes, and until recently generative models for them worked directly on continuous atomic coordinates. A newer line of work instead converts 3D backbones into discrete tokens — short categorical symbols drawn from a learned codebook, each summarizing a local piece of geometry — so that ordinary text Transformers can generate protein structures the way they generate sentences. In a separate literature, researchers edit images and language representations by simple addition and subtraction in latent space: compute a "direction" for a concept, add it to a latent code, and decode. This works surprisingly often, but only when the arithmetic is confined to a carefully isolated subspace; done in the raw embedding space, the result drifts off the data manifold into nonsense. Finally, whenever two data types such as text and images are projected into one shared embedding space, their embeddings refuse to mix: they settle into two separate cones separated by a measurable void, the cross-modal representation gap, and naive cross-modal arithmetic falls straight into that void.
This proposal connects all three. The plan is to build a joint text-and-structure model in which proteins are represented as structural token sequences, decompose the resulting text–structure embedding space into shared ("bimodal") and modality-specific ("unimodal") concept atoms using a sparse-autoencoder technique proven on vision–language models, and then perform closed-form editing: take the latent of an existing protein's structural tokens, add a text-derived direction restricted to the shared subspace — "add more beta sheet," "make the helices longer" — and decode a physically viable, predictably deformed 3D fold. No paper in the surveyed literature does this: the subspace-restricted arithmetic has only been shown for retrieving 2D images, and all text-driven protein editing so far operates on one-dimensional amino acid sequences, never on structural tokens.
Core Claim
The claim has two parts, one diagnostic and one constructive. Diagnostically: once 3D structure is discretized into tokens, the text–structure modality gap should decompose the same way the image–text gap does — into a small set of high-energy unimodal bias atoms (geometric properties with no natural-language counterpart, such as local frame handedness or packing chirality) and bimodal atoms carrying the alignment signal. Cross-Modal Redundancy and the Geometry of Vision–Language Embeddings proved this decomposition for six frozen vision–language dual encoders, showing that filtering the unimodal atoms collapses the cone separation without hurting retrieval; nobody has checked whether the 3D-to-text gap has the same anatomy, and the surveyed structural-tokenization work motivates the question without answering it — 3D-MolT5 reports that bolting 3D features onto a text model leaves the modalities not speaking the same language, and ProteinDT had to patch its residual gap with an ad-hoc feedforward "facilitator" network.
Constructively: if the decomposition holds, then arithmetic restricted to the bimodal subspace becomes a zero-shot structure editor. Concretely, with $z$ the latent of a protein's structural-token sequence, and $\Delta z = E_{\text{text}}(\text{prompt}) - E_{\text{text}}(\text{neutral})$ a direction computed from two text encodings, the edit is
$$z' = z + P\,\Delta z,$$
where $P$ projects onto the span of bimodal atoms, and $z'$ is decoded into structural tokens and then coordinates. This is one matrix multiply and one decode — no per-edit gradient optimization (unlike ProteinDT's latent interpolation and optimization loops) and no paired (structure, instruction, edited structure) training data (unlike any end-to-end editing model).
Research Program
The first component is the tokenizer and decoder, which we reuse rather than invent. SaProt provides per-residue structural tokens aligned with sequence, which is exactly the granularity needed to pair text descriptions with structural token sequences; the InstaDeep vector-quantized autoencoder protein-structure-tokenizer from Learning the Language of Protein Structure offers a learned codebook alternative with 4,096 to 64,000 tokens. For decoding latent edits back into geometry we adopt the SaDiT design — a Diffusion Transformer operating on the continuous latents of structural tokens — because SaDiT already showed those latents organize by high-level topology (CATH fold class), which is a prerequisite for topology-level text directions to exist as linear directions at all. Where a fully discrete path is needed we fall back to an autoregressive decoder over tokens, as in LeFlur, which showed a standard text Transformer can fold, inverse-fold, and generate from a unified token stream.
The second component is the dual encoder and its decomposition. We train a text encoder and a structural-token-sequence encoder with contrastive alignment on text–structure pairs derived from SwissProtCLAP's 441,013 Swiss-Prot annotation pairs by attaching predicted structures to each entry — mirroring ProteinDT's data regime but with structural tokens replacing amino-acid sequences. On the frozen aligned embeddings we train an Aligned Sparse Autoencoder, the Matching-Pursuit SAE with an equal-activation-energy alignment penalty from Cross-Modal Redundancy, yielding bimodal and unimodal atoms. This is the method-supplying citation: we reuse their machinery unchanged and change only the modality on the structure side. The open scientific question is what the unimodal atoms turn out to encode — our hypothesis is geometric properties genuinely absent from language (frame chirality, local packing density), which would explain why Learning Robust Visual-Semantic Embeddings found instance-level contrastive losses leave global cone separation intact, and would give the facilitator network of ProteinDT a mechanistic explanation it currently lacks.
The third component is the edit operator itself, plus two structural analyses the literature flags as unresolved. One is the tension between invariant tokenization and directional arithmetic: SaProt and SaDiT insist on rotation-and-translation-invariant tokens to suppress coordinate noise, while Kanzi showed dropping invariance at 30M parameters still reconstructs well and enables spatial conditioning. Since text directions should themselves be orientation-agnostic, we test whether invariant tokens actually protect the arithmetic or whether the projection $P$ already absorbs orientation. The other is codebook capacity: LeFlur showed 256 tokens suffice for reconstruction, but quantization boundaries may shatter continuous linear trajectories into step functions; we measure arithmetic quality as a function of codebook size from 256 tokens through 64,000 to fully continuous latents.
Proposed Experiments
Experiment 1 measures the geometry of the text–structure gap. On held-out Swiss-Prot text–structure retrieval, we report retrieval recall and an out-of-distribution score for three variants: raw contrastive embeddings, a mean-shift (cone-centering) correction, and SAE-A unimodal-atom filtering. The reference pattern comes from the vision–language result: on LAION, filtering preserved recall at 0.760 where simple shifting dropped it to 0.704, and on FashionIQ unconstrained arithmetic scored 0.97 OOD versus 0.63 in the bimodal subspace with recall essentially unchanged (0.681 versus 0.683). No reported result exists on Lacuna for any 3D-structure-to-text modality-gap decomposition, which is precisely the gap this experiment fills.
Experiment 2 is the editing benchmark. We construct edit pairs from CATH 4.3 using SaDiT's splits (28,432 training, 2,100 validation, 1,200 test structures, chains 50–512 residues) and Swiss-Prot descriptions: an edit is defined by a source structure, a target structure differing in one annotated topological attribute, and a text direction computed from the attribute's description. Three metrics: decode viability, measured as designability by self-consistency (the fraction of decodes whose generated structure matches its own sequence's predicted fold), where SaDiT's 99.5% unconditional designability sets the ceiling a healthy decoder inherits; edit effectiveness, the measured change in the intended property (secondary-structure fractions via DSSP, helix length, sheet count); and preservation, the TM-score of unedited regions against the source. The decisive comparison is arithmetic with versus without the bimodal projection $P$: if the projection matters, we expect the raw-space variant to decode to unviable or off-target structures, reproducing in 3D the failure mode Cross-Modal Redundancy documented in 2D.
Experiment 3 is the capacity ablation: identical pipeline, codebooks of 256 (LeFlur-style SLQ), 4,096 and 64,000 (InstaDeep VQ), and continuous SaDiT latents. The metric is a linearity curve — fraction of viably decoded structures at fractional step sizes $t\,\Delta z$ for $t \in [0,1]$ — which directly tests whether discrete codebooks break the parallelogram structure that Vector Arithmetic in Concept and Token Subspaces recovered in language models (roughly 80% analogy accuracy inside a rank-256 concept subspace versus 47% in the raw hidden state).
Baselines
The do-nothing reference decodes the unedited structural latent: it should score perfectly on preservation and zero on edit effectiveness, and any editing method must at minimum match its decode viability, since a decoder that produces junk when fed its own latents cannot support arithmetic at all. For context on what a healthy structural-token decoder achieves, SaDiT reports 99.5% unconditional designability and LeFlur reports an 84.8% pass rate at RMSD under 2 Å on de novo backbone generation; our edited decodes must land near these numbers, not merely "look like proteins."
The first real baseline is unconstrained vector arithmetic in the raw contrastive space — the same recipe with $P$ set to the identity. This is the strongest ablation for isolating the proposal's key ingredient, because it differs from the full method by exactly one projection, and the vision–language evidence predicts it fails (OOD 0.97 versus 0.63 on FashionIQ). If it does not fail in 3D, the core claim about needing the subspace is wrong and we will report that.
The strongest existing competitor is ProteinDT, the only published system doing text-driven latent editing of proteins. It aligns sequence and text encoders with ProteinCLAP contrastive training, bridges the residual gap with the ProteinFacilitator network (essential, improving retrieval by up to 50% over direct decoding from text embeddings), and edits via per-sample latent interpolation or iterative latent optimization before decoding amino-acid sequences — reported at over 90% retrieval accuracy of generated sequences back to their prompts. We run it on our benchmark by decoding its sequences and folding them, and we also run its optimization procedure inside our structural latent space, so the comparison isolates the representation (structural tokens versus sequence) and the edit operator (closed-form arithmetic versus iterative optimization) rather than the data. Both systems get the same SwissProtCLAP-derived pairs, the same decoder family, and the same sampling compute.
A second strong baseline is ProtET's conditioning recipe: Feature-wise Linear Modulation, where text-instruction embeddings are fused into the generator's activations, trained on over 67 million protein–biotext pairs and reported to improve sequence stability by roughly 17%. We train a FiLM-conditioned version of our structural-token decoder on our edit pairs with the same compute and tuning budget. This baseline is the honest end-to-end alternative, and if it beats subspace arithmetic on edit effectiveness while matching viability, the practical case for arithmetic weakens to its zero-shot advantage on attributes never seen in training — which we test directly by holding out attributes.
Finally, a gap-closure ablation replaces atom filtering with mean-shift centering, since Cross-Modal Redundancy showed shifting recovers geometry worse (LAION recall 0.704 versus 0.760).
Why not just…?
Why not just condition a continuous diffusion model on text? Continuous SE(3) diffusion models such as RFdiffusion remain the quality leaders, and the Kanzi authors concede token-based models do not yet match them (their 250M-parameter autoregressive model reaches a 0.617 designability score with scRMSD of 3.655 Å). This is a real weakness and we will not hide it. But SaDiT shows continuous coordinate diffusion costs 168 seconds per sample against 0.73 seconds over structural token latents — a 230-fold difference — and collapses on long chains where token models hold 75% designability at 800 residues. More fundamentally, a conditioned diffusion model answers "generate something matching this prompt," not "edit this specific structure by this semantic delta," and offers no closed-form edit algebra to study. Our question is about the geometry of the gap and the existence of linear text-to-structure directions; a diffusion conditioner cannot answer it either way.
Why not just train an end-to-end editing model with cross-attention or FiLM? This is the most serious alternative, and we treat it as a head-to-head baseline rather than an argument. Its cost is data: paired (structure, instruction, edited structure) triplets essentially do not exist, and ProtET needed 67 million pairs to make sequence-level instruction tuning work, while arithmetic needs only unpaired text. Our decisive test holds out entire attributes (for example, all sheet-curvature edits) from the FiLM baseline's training set and applies the arithmetic to them zero-shot; if FiLM generalizes to held-out attributes as well as arithmetic does, we will conclude the subspace machinery is unnecessary and say so in the paper.
Why not just do more contrastive training and add a facilitator network? Because the evidence says contrastive alignment alone leaves the global cones intact: Learning Robust Visual-Semantic Embeddings shows instance-level InfoNCE pulls pairs together locally while distributions stay separated, and ProteinDT found direct decoding from contrastively aligned text embeddings failed even after 441,013 pairs, requiring the facilitator. The facilitator works but is unexplained and adds a trained module per modality pair. Our claim is that unimodal-atom filtering is a principled replacement: Experiment 1 runs filtering and a trained facilitator side by side on the same embeddings, and if filtering matches the facilitator's retrieval while also enabling arithmetic the facilitator cannot, the decomposition earns its place; if the facilitator wins, that itself characterizes what the linear-atom model misses in 3D.
Decisive Checks
The proposal succeeds if three things are observed together: the text–structure gap decomposes into bimodal and unimodal atoms with the unimodal atoms provably geometric rather than linguistic; bimodal-restricted arithmetic decodes to viable structures (designability near the SaDiT-family ceiling) with measurable, correctly signed property changes where raw-space arithmetic decodes to junk; and the linearity curve degrades gracefully with codebook size rather than collapsing below some capacity threshold. It fails informatively if the 3D gap does not decompose like the 2D one — that result would itself constrain every future attempt to bolt text onto structural tokenizers, and we would report it as such.
References
- Cross-Modal Redundancy and the Geometry of Vision–Language Embeddings
- 3D-MolT5: Leveraging Discrete Structural Information for Molecule-Text Modeling
- A Text-guided Protein Design Framework
- SaProt: Protein Language Modeling with Structure-aware Vocabulary
- Learning the Language of Protein Structure
- SaDiT: Efficient Protein Backbone Design via Latent Structural Tokenization and Diffusion Transformers
- LeFlur: A Biomolecular Design Model with Latent Structure Tokens
- Learning Robust Visual-Semantic Embeddings
- Flow Autoencoders are Effective Protein Tokenizers
- Vector Arithmetic in Concept and Token Subspaces