HIGHT: Hierarchical Graph Tokenization for Graph-Language Alignment

Yongqiang Chen, Quanming Yao, Juzheng Zhang, James Cheng, Yatao Bian

Introduction

Large Language Models (LLMs) have shown remarkable potential in cross-disciplinary applications, yet granting them "chemical eyes" remains a significant challenge. Most current Large Graph-Language Models (LGLMs) represent molecules as flat collections of atoms (nodes). This approach ignores the hierarchical nature of chemistry, where functional groups or "motifs"—such as hydroxide (-OH) or carboxylic acid (-COOH) groups—dictate a molecule's behavior. By failing to explicitly encode these higher-order structures, models often struggle with structural alignment, leading to "hallucinations" where the model incorrectly identifies functional groups that are not present.

The HIGHT Framework

To address these limitations, the authors introduce HIGHT (HIerarchical GrapH Tokenization), a framework designed to bridge the gap between molecular graphs and natural language by explicitly modeling hierarchical structures.

Overview of the HIGHT framework and its performance across various biochemical tasks.

Hierarchical Tokenization Strategy

HIGHT employs a multi-level tokenization process:

  1. Motif Decomposition: Using the BRICS algorithm, molecules are decomposed into chemically meaningful substructures.
  2. Discrete Token Generation: Separate Vector Quantized-Variational AutoEncoders (VQVAEs) generate discrete tokens for individual atoms, motifs, and the global molecular structure.
  3. Structural Context: To preserve the spatial relationship between these components, the framework utilizes Laplacian positional embeddings, providing the LLM with the structural context missing from flat tokenization methods.

Two-Stage Instruction Tuning

The model undergoes a rigorous training process. In the first stage (Alignment), HIGHT is pretrained on the newly introduced HiPubChem dataset—comprising 295k molecule-text pairs—to map graph tokens into the LLM's embedding space. In the second stage (Task-Specific), the model is fine-tuned using Low-Rank Adaptation (LoRA) on specialized downstream tasks.

Combating the "Yes-Man" Problem

A critical insight of this research is the identification of the "Yes-Man" problem in existing LGLMs. When queried about the presence of specific functional groups, traditional node-centric models tend to answer affirmatively regardless of the actual molecular structure.

Comparison between node-centric tokenization (left), which leads to hallucinations, and HIGHT tokenization (right), which correctly identifies motifs.

By making motifs explicit tokens, HIGHT provides the model with direct evidence of functional groups. This architectural shift, combined with "hierarchical" text descriptions in the training data, allows the model to accurately verify structural features rather than relying on implicit inference.

Experimental Results

HIGHT was evaluated across 14 diverse biochemical benchmarks, demonstrating superior performance in several key areas:

  • Hallucination Reduction: On the MotifHallu diagnostic benchmark, HIGHT reduced functional group hallucinations by up to 40% compared to state-of-the-art models like GIMLET and InstructMol.
  • Property Prediction: In quantum chemical property regression (QM9 dataset), HIGHT achieved a Mean Absolute Error (MAE) of 0.0086, significantly outperforming the previous baseline of 0.0130.
  • Chemical Synthesis: The model showed substantial gains in predicting reaction reagents, forward reaction outcomes, and retrosynthetic pathways, confirming that motif-level understanding is essential for modeling chemical transformations.
  • Molecular Captioning: HIGHT produced more precise textual descriptions, improving BLEU-4 scores on the ChEBI-20 dataset from 0.381 to 0.405.

Conclusion and Future Directions

HIGHT represents a significant advancement in graph-language alignment by moving beyond atom-level representations. While the current framework focuses on 2D connectivity, future work may incorporate 3D spatial coordinates to better model complex biological interactions. Additionally, the hierarchical tokenization approach holds promise for other domains, such as social network analysis or knowledge graph reasoning, where structural motifs carry high semantic weight.

Summary updated

BibTeX
@inproceedings{chen2025hight,
  author = {Yongqiang Chen and Quanming Yao and Juzheng Zhang and James Cheng and Yatao Bian},
  title = {{HIGHT: Hierarchical Graph Tokenization for Graph-Language Alignment}},
  booktitle = {International Conference on Machine Learning},
  year = {2025},
  url = {https://openreview.net/forum?id=wpbNczwAwV},
}
Figure