CrysGNN : Distilling pre-trained knowledge to enhance property prediction for crystalline materials

Kishalay Das, Bidisha Samanta, Pawan Goyal, Seung-Cheol Lee, Satadeep Bhattacharjee, Niloy Ganguly

Abstract

In recent years, Graph Neural Network (GNN) based approaches have emerged as a powerful technique to encode the complex topological structure of crystal materials into enriched representation spaces. However, these supervised models typically require vast amounts of property-tagged data, which is often unavailable for many material properties. To address this, we present CrysGNN, a pre-trained GNN framework that captures both node and graph-level structural information using a massive dataset of 800,000 unlabeled crystal structures. By extracting distilled knowledge from CrysGNN and injecting it into state-of-the-art (SOTA) property predictors, we significantly enhance their accuracy. Our experiments demonstrate that this distillation process outperforms conventional fine-tuning, allowing existing models to surpass their vanilla performance across various benchmarks.

Introduction

Predicting the physical and chemical properties of crystals—such as bandgap, formation energy, and bulk modulus—is fundamental to the discovery of new materials for energy storage, electronics, and catalysis. Traditionally, researchers have relied on Density Functional Theory (DFT) to calculate these properties. While accurate, DFT is computationally expensive and scales poorly with system size.

Deep learning, particularly Graph Neural Networks (GNNs), offers a promising alternative by treating atoms as nodes and chemical bonds as edges. However, GNNs are notoriously data-hungry. While the scientific community has access to millions of known crystal structures (unlabeled data), the subset of materials with high-quality property labels is relatively small. CrysGNN bridges this gap by leveraging unlabeled data to learn the "DNA" of crystals before specializing in specific property prediction tasks.

The CrysGNN Framework

CrysGNN utilizes a multi-level self-supervised learning strategy to build a robust understanding of crystalline structures without requiring property labels.

1. Multi-Graph Representation

Crystals are modeled as multi-graphs to capture their periodic nature. Nodes represent atoms, while edges represent chemical bonds, ensuring the model understands the spatial arrangement and connectivity of the lattice.

2. Node-Level Pre-training

The model is trained to reconstruct atomic features (such as electronegativity) and local connectivity. By predicting which atoms are neighbors and their specific attributes, the GNN learns the fundamental building blocks of chemical environments.

3. Graph-Level Pre-training and Symmetry

Symmetry is a defining characteristic of crystals. CrysGNN incorporates this by:

  • Space Group Classification: Learning to categorize crystals into one of the 230 space groups.
  • Contrastive Learning: Ensuring that crystals within the same Crystal System (e.g., Cubic, Hexagonal) have similar mathematical representations in the latent space.

4. Knowledge Distillation: The "Plug-and-Play" Booster

The core innovation of this work is the use of knowledge distillation rather than traditional fine-tuning. In standard fine-tuning, a pre-trained model is modified for a specific task. In the CrysGNN framework, the pre-trained model acts as a "teacher." Other SOTA models (the "students") are trained to match the intermediate representations of CrysGNN. This allows the student models to inherit structural wisdom without requiring them to share the same architecture as the teacher, making CrysGNN a universal booster for existing algorithms.

Experimental Setup and Datasets

To train CrysGNN, we curated a massive dataset of 800,000 unlabeled crystal graphs sourced from the Open Quantum Materials Database (OQMD) and the Materials Project (MP).

For evaluation, we tested the framework on several benchmark datasets:

  • MP 2018.6.1: 69,000 materials focusing on bandgap and formation energy.
  • JARVIS-DFT: 55,000 materials covering 19 different properties.
  • OQMD-EXP: A critical dataset of 1,500 experimental data points used to evaluate the model's ability to overcome the systematic biases inherent in DFT-calculated data.

Key Results and Insights

Performance Gains Across SOTA Models

We integrated CrysGNN with four leading models: CGCNN, GATGNN, CrysXPP, and ALIGNN. In every instance, the distilled version outperformed the vanilla version:

  • CGCNN: 16.20% average improvement.
  • CrysXPP: 12.21% average improvement.
  • GATGNN: 8.02% average improvement.
  • ALIGNN: 4.19% average improvement.

Data Efficiency

One of the most significant findings was the framework's data efficiency. Distilled models achieved higher accuracy with only 20% of the training data than vanilla models achieved with 60%. This makes CrysGNN particularly valuable for rare materials or properties where experimental data is extremely scarce.

Mitigating DFT Bias

Most AI models in material science are trained on synthetic DFT data, which can differ from real-world experimental results. CrysGNN showed superior performance on the OQMD-EXP dataset, suggesting that its structural pre-training helps the model generalize better to physical reality than models trained solely on calculated labels.

Limitations and Future Work

While CrysGNN provides a substantial boost, the improvements were more modest for highly complex models like ALIGNN, which already incorporates three-body interactions (bond angles). This suggests that future iterations of the CrysGNN teacher may need to explicitly model angular information to provide better guidance to advanced student architectures.

Additionally, while the model excels at structure-dependent properties, its impact on purely electronic properties—which depend more on fine-grained electronic descriptions than atomic arrangement—is less pronounced. Future research will explore multi-modal pre-training, potentially incorporating text-based chemical descriptions or spectral data to further enrich the learned representations.

Conclusion

CrysGNN demonstrates that the vast amount of unlabeled crystal data available today can be effectively harnessed to improve property prediction. By moving beyond simple fine-tuning to a distillation-based approach, we provide a flexible, architecture-agnostic tool that enhances the accuracy and data efficiency of the entire material science machine learning ecosystem.

Summary updated

Abstract

In recent years, graph neural network (GNN) based approaches have emerged as a powerful technique to encode complex topological structure of crystal materials in an enriched representation space. These models are often supervised in nature and using the property-specific training data, learn relationship between crystal structure and different properties like formation energy, bandgap, bulk modulus, etc. Most of these methods require a huge amount of property-tagged data to train the system which may not be available for different properties. However, there is an availability of a huge amount of crystal data with its chemical composition and structural bonds. To leverage these untapped data, this paper presents CrysGNN, a new pre-trained GNN framework for crystalline materials, which captures both node and graph level structural information of crystal graphs using a huge amount of unlabelled material data. Further, we extract distilled knowledge from CrysGNN and inject into different state of the art property predictors to enhance their property prediction accuracy. We conduct extensive experiments to show that with distilled knowledge from the pre-trained model, all the SOTA algorithms are able to outperform their own vanilla version with good margins. We also observe that the distillation process provides a significant improvement over the conventional approach of finetuning the pre-trained model. We have released the pre-trained model along with the large dataset of 800K crystal graph which we carefully curated; so that the pretrained model can be plugged into any existing and upcoming models to enhance their prediction accuracy.
BibTeX
@inproceedings{das2023crysgnn,
  author = {Kishalay Das and Bidisha Samanta and Pawan Goyal and Seung-Cheol Lee and Satadeep Bhattacharjee and Niloy Ganguly},
  title = {{CrysGNN : Distilling pre-trained knowledge to enhance property prediction for crystalline materials}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  url = {https://openreview.net/forum?id=jqhyzgv7gD},
}
Figure