TL;DR
Cuttlefish is a new multimodal framework that allows Large Language Models (LLMs) to reason across diverse biomolecular structures—including proteins, DNA, RNA, and small molecules—using a unified "all-atom" approach. By implementing Scaling-Aware Patching, the model adaptively allocates more tokens to complex structures while using a Geometry Grounding Adapter to inject explicit 3D spatial cues, significantly reducing structural hallucinations compared to fixed-length or sequence-only models.
What Problem Does This Paper Address?
Current AI models for biology often suffer from two major bottlenecks:
- Modality Silos: Most models are specialized for just one type of molecule (e.g., only proteins or only small molecules).
- Rigid Tokenization: Existing "structure-aware" LLMs typically use sequence-style encodings (like SMILES strings) or a fixed number of "connector tokens" (like a Q-Former).
- Sequence-only inputs lack 3D geometric context, leading the model to "hallucinate" chemical properties that aren't physically possible.
- Fixed-length connectors create a "compression bottleneck." If a model uses 64 tokens for every input, it over-allocates resources to tiny molecules (diluting attention) and over-compresses massive proteins (losing critical detail).
The authors aim to create a unified, geometry-grounded interface that scales its computational budget based on the actual complexity of the biological entity.
Methods
The authors introduce Cuttlefish, which uses a three-part architecture to bridge microscopic geometry with linguistic reasoning:
- SE(3)-Equivariant Encoder: An Equivariant Graph Neural Network (EGNN) processes all-atom inputs (coordinates and features) to ensure the model's understanding remains stable regardless of how a molecule is rotated or translated in 3D space.
- Scaling-Aware Patching: This is the core innovation. Instead of a fixed token count, Cuttlefish uses an instruction-conditioned gating policy. It identifies "anchor atoms" based on the user's prompt and grows variable-sized "soft patches" around them. This allows the token budget to grow naturally with the size and complexity of the structure.
- Geometry Grounding Adapter: This module refines the patch tokens through cross-attention with the full atom-level embeddings. It then injects these "geometry-informed" tokens directly into the LLM’s embedding space, ensuring the language output is physically grounded in the 3D structure.
The training involves a two-stage process: first, aligning the structural tokens with the text (modality alignment), and then fine-tuning the LLM backbone to accommodate the new structural data.
Datasets
The researchers evaluated Cuttlefish on several established benchmarks:
- Mol-Instructions: A large-scale biomolecular instruction dataset.
- DNA-Chat & RNA-QA: Specialized datasets for nucleic acid reasoning.
Crucially, they introduced a new dataset called GEO-AT. This is the first unified all-atom instruction dataset that includes 3D coordinates and language annotations for molecules, proteins, and nucleic acids. It serves as a "community resource" to catalyze the development of geometry-grounded LLMs.
Key Insights and Ideas
- Instruction-Conditioned Selection: Unlike previous models that compress a molecule the same way every time, Cuttlefish changes which parts of the structure it "focuses" on based on the question asked.
- The "Aha!" of Mixed-Modality Training: The authors found that training on molecules, proteins, and DNA/RNA simultaneously actually improved performance. The model benefits from "cross-modality diversity," learning general chemical relations that transfer across different types of biological entities.
- Ending the Compression Bottleneck: By allowing the number of tokens to scale with the number of atoms (as seen in Fig. 6), Cuttlefish maintains stable performance on large proteins where fixed-capacity models (like Mol-Llama) typically see a sharp drop in accuracy.
Key Results
- Reduced Hallucinations: In a functional group hallucination test, Cuttlefish achieved the lowest hallucination rate (0.07 for molecules, 0.04 for proteins) compared to strong baselines like Mol-Llama and ProtChatGPT.
- Superior Reasoning: Across 17 tasks in Mol-Instructions and 20 tasks in DNA-Chat, Cuttlefish consistently outperformed modality-specific models. For example, it achieved a METEOR score of 0.715 on molecular captioning, surpassing the previous state-of-the-art.
- Zero-Shot Entity Recognition: Even when given only structural coordinates without any text hints, Cuttlefish could accurately identify whether it was looking at a small molecule, a protein, or a nucleic acid.
Limitations and Future Work
- Extremely Large Entities: While Cuttlefish scales better than previous models, it still struggles with "extremely large" entities (e.g., >50,000 nodes). The authors suggest a "Retrieval-Augmented Hierarchical Reasoning" approach for future work to handle entire cellular complexes.
- Upstream Noise: The model's performance is highly sensitive to the quality of the input 3D coordinates. If the provided structure is poorly resolved, the model's reasoning suffers.
- Coarse Supervision: Currently, the model relies on the general reasoning capabilities of the underlying LLM (like Llama-3). The authors hope to build more specialized "chain-of-thought" datasets for all-atom reasoning to improve deep scientific deliberation.
Summary updated