Crystals with Transformers on Graphs, for Prediction of Unconventional Crystal Material Properties and the Benchmark

Hongyi Wang, Ji Sun, Jinzhe Liang, Li Zhai, Zitian Tang, Zijian Li, Wei Zhai, Xusheng Wang, Weihao Gao, Sheng Gong, Bolong Huang, Hua Zhang

Abstract

Predicting the properties of crystalline materials is a fundamental challenge in materials science. While Graph Neural Networks (GNNs) excel at capturing local atomic environments, they often struggle with the long-range structural orders characteristic of complex materials like Metal-Organic Frameworks (MOFs) and 2D layers. We present CrysToGraph, a hybrid geometric architecture that integrates local graph convolutions with global transformers. To evaluate this approach, we introduce UnconvBench, a benchmark focused on "unconventional" materials. CrysToGraph achieves state-of-the-art (SOTA) performance on 10 out of 15 diverse datasets, demonstrating superior capability in modeling both short-range bonds and macroscopic structural patterns.

Introduction

The accurate prediction of crystal properties, such as band gaps or formation energies, is critical for accelerating material discovery. Traditional models primarily utilize GNNs, representing atoms as nodes and bonds as edges. However, standard message-passing GNNs are limited by their receptive field, often failing to capture the long-range order or complex porous geometries found in unconventional crystals. In systems like MOFs or defected crystals, macroscopic properties are frequently dictated by patterns repeating over large distances, necessitating a model that can "see the forest for the trees."

Methodology: The CrysToGraph Architecture

CrysToGraph employs a "local-then-global" hierarchical approach to represent crystal structures effectively.

Local Interactions: eTGC Block

The edge-engaged Transformer Graph Convolution (eTGC) handles short-range interactions. It utilizes a transformer-based message-passing mechanism to update both node (atom) and edge (bond) features. To capture critical geometric information like bond angles (three-body interactions), the model incorporates line graphs.

Transformation of a crystal graph G to its corresponding line graph L(G) to model bond angles.

Global Dependencies: GwT Block

Following the local processing, the Graph-wise Transformer (GwT) serves as the global engine. By treating all atoms in a unit cell as a sequence and applying self-attention, the GwT captures dependencies between distant atoms that standard GNNs would miss. To preserve spatial topology within the order-agnostic transformer, Laplacian and Random Walk positional encodings are utilized.

The CrysToGraph architecture: eTGC blocks capture local chemistry, followed by GwT blocks for global symmetry, concluding with a Feed-Forward Neural Network (FFNN).

UnconvBench and Evaluation

The authors introduce UnconvBench, a curated suite of 11 datasets designed to test models on unconventional systems, including 2D materials (exfoliation energy), MOFs (CO2 adsorption), and defected crystals. The model was also benchmarked against MatBench, the industry standard for traditional bulk crystals.

Key Findings

  • Structural Hierarchy: Stacking local layers before global layers (eTGC → GwT) proved most effective, mirroring the physical reality where local bonding forms the basis for global structure.
  • Scalability: While local GNNs suffice for small, ordered crystals, the global transformer becomes the dominant factor in accuracy as unit cell size increases, particularly in MOFs.
  • Geometric Nuance: Explicitly modeling bond angles via line graphs is vital for defected crystals, where broken symmetries make local geometric details more critical.

Results

CrysToGraph achieved SOTA results on 10 out of 15 datasets. Notably, on the qmof dataset, it significantly outperformed established models like ALIGNN and Matformer. For complex engineering tasks such as CO2 adsorption, the model reached an $r^2$ score of 0.8946, highlighting its practical utility.

Conclusion

CrysToGraph successfully bridges local and global structural representations in crystal property prediction. By combining edge-engaged convolutions with graph-wise transformers, the model sets a new standard for both traditional and unconventional materials. Future work may extend this architecture to molecular dynamics and other complex chemical systems where the interplay of local and global scales is paramount.

Summary updated

BibTeX
@inproceedings{wang2025crystals,
  author = {Hongyi Wang and Ji Sun and Jinzhe Liang and Li Zhai and Zitian Tang and Zijian Li and Wei Zhai and Xusheng Wang and Weihao Gao and Sheng Gong and Bolong Huang and Hua Zhang},
  title = {{Crystals with Transformers on Graphs, for Prediction of Unconventional Crystal Material Properties and the Benchmark}},
  booktitle = {Battery},
  year = {2025},
  doi = {10.1002/bte2.70004},
}
Figure