Introduction
Large Language Models (LLMs) have shown remarkable potential in cross-disciplinary applications, yet granting them "chemical eyes" remains a significant challenge. Most current Large Graph-Language Models (LGLMs) represent molecules as flat collections of atoms (nodes). This approach ignores the hierarchical nature of chemistry, where functional groups or "motifs"—such as hydroxide (-OH) or carboxylic acid (-COOH) groups—dictate a molecule's behavior. By failing to explicitly encode these higher-order structures, models often struggle with structural alignment, leading to "hallucinations" where the model incorrectly identifies functional groups that are not present.
The HIGHT Framework
To address these limitations, the authors introduce HIGHT (HIerarchical GrapH Tokenization), a framework designed to bridge the gap between molecular graphs and natural language by explicitly modeling hierarchical structures.
Hierarchical Tokenization Strategy
HIGHT employs a multi-level tokenization process:
- Motif Decomposition: Using the BRICS algorithm, molecules are decomposed into chemically meaningful substructures.
- Discrete Token Generation: Separate Vector Quantized-Variational AutoEncoders (VQVAEs) generate discrete tokens for individual atoms, motifs, and the global molecular structure.
- Structural Context: To preserve the spatial relationship between these components, the framework utilizes Laplacian positional embeddings, providing the LLM with the structural context missing from flat tokenization methods.
Two-Stage Instruction Tuning
The model undergoes a rigorous training process. In the first stage (Alignment), HIGHT is pretrained on the newly introduced HiPubChem dataset—comprising 295k molecule-text pairs—to map graph tokens into the LLM's embedding space. In the second stage (Task-Specific), the model is fine-tuned using Low-Rank Adaptation (LoRA) on specialized downstream tasks.
Combating the "Yes-Man" Problem
A critical insight of this research is the identification of the "Yes-Man" problem in existing LGLMs. When queried about the presence of specific functional groups, traditional node-centric models tend to answer affirmatively regardless of the actual molecular structure.
By making motifs explicit tokens, HIGHT provides the model with direct evidence of functional groups. This architectural shift, combined with "hierarchical" text descriptions in the training data, allows the model to accurately verify structural features rather than relying on implicit inference.
Experimental Results
HIGHT was evaluated across 14 diverse biochemical benchmarks, demonstrating superior performance in several key areas:
- Hallucination Reduction: On the MotifHallu diagnostic benchmark, HIGHT reduced functional group hallucinations by up to 40% compared to state-of-the-art models like GIMLET and InstructMol.
- Property Prediction: In quantum chemical property regression (QM9 dataset), HIGHT achieved a Mean Absolute Error (MAE) of 0.0086, significantly outperforming the previous baseline of 0.0130.
- Chemical Synthesis: The model showed substantial gains in predicting reaction reagents, forward reaction outcomes, and retrosynthetic pathways, confirming that motif-level understanding is essential for modeling chemical transformations.
- Molecular Captioning: HIGHT produced more precise textual descriptions, improving BLEU-4 scores on the ChEBI-20 dataset from 0.381 to 0.405.
Conclusion and Future Directions
HIGHT represents a significant advancement in graph-language alignment by moving beyond atom-level representations. While the current framework focuses on 2D connectivity, future work may incorporate 3D spatial coordinates to better model complex biological interactions. Additionally, the hierarchical tokenization approach holds promise for other domains, such as social network analysis or knowledge graph reasoning, where structural motifs carry high semantic weight.
Summary updated