CrysGNN: Distilling Pre-trained Knowledge to Enhance Property Prediction for Crystalline Materials

Kishalay Das, Bidisha Samanta, Pawan Goyal, Seung-Cheol Lee, Satadeep Bhattacharjee, Niloy Ganguly

TL;DR

This paper introduces CrysGNN, a novel pre-training framework for Graph Neural Networks (GNNs) that learns structural and chemical information from a massive dataset of unlabeled crystalline materials. By distilling this learned knowledge into existing state-of-the-art property prediction models, CrysGNN significantly boosts their accuracy, outperforming traditional fine-tuning methods and demonstrating a path towards more efficient and accurate materials discovery.

What Problem Does This Paper Address?

Predicting the properties of crystalline materials (like their bandgap or formation energy) is crucial for discovering new materials with desired functionalities. While Graph Neural Networks (GNNs) have shown great promise in encoding the complex structures of these materials, they typically require vast amounts of labeled data for each specific property. This labeled data is often scarce and expensive to obtain, especially for less common properties. The paper addresses this "data sparsity" problem by leveraging the abundance of unlabeled crystal structure data. The challenge is to extract meaningful, generalizable knowledge from this unlabeled data that can then be effectively transferred to improve the performance of models trained on limited labeled data for specific property prediction tasks.

Methods

The core of CrysGNN is a self-supervised pre-training framework built upon a GNN encoder-decoder architecture.

  • Crystal Graph Representation: Crystalline materials are represented as multi-graphs, where atoms are nodes and bonds are edges. Node features include chemical properties, and edge weights represent bond lengths.
  • Self-Supervised Pre-training: CrysGNN is trained end-to-end using two main self-supervised objectives:
  • Node-Level Decoding: The model reconstructs node features (chemical properties of atoms) and local connectivity (atom-atom relationships and bond strengths) from node embeddings. This helps the model learn local chemical semantics.
  • Graph-Level Decoding: This involves two components:
  • Space Group Prediction: The model predicts the space group of a crystal, which describes its symmetry.
  • Contrastive Learning: The model learns to distinguish between crystal graphs belonging to the same crystal system (positive pairs) and those from different systems (negative pairs) in an embedding space. This captures global structural similarities.
  • Knowledge Distillation: After pre-training, CrysGNN's learned knowledge is "distilled" into existing state-of-the-art (SOTA) property predictors. Instead of simply fine-tuning the pre-trained model, knowledge distillation transfers the learned representations and relationships. This is achieved by minimizing a combined loss that includes the standard property prediction loss and a knowledge distillation loss, which encourages the property predictor's intermediate embeddings to match those from the pre-trained CrysGNN.

Datasets

  • Pre-training Dataset: A large, newly curated dataset of 800,000 unlabeled crystal graphs was created by combining data from the Materials Project (MP) and OQMD databases.
  • Downstream Task Evaluation Datasets:
    • Materials Project (MP) 2018.6.1: Used for evaluating property prediction on a subset of the pre-training data.
  • JARVIS-DFT: A separate dataset used for evaluating property prediction, ensuring the model generalizes to unseen data.
  • OQMD-EXP: A small dataset of 1,500 experimental data points for formation energy, used to investigate the removal of DFT error bias.

Key Insights and Ideas

  • Leveraging Unlabeled Data is Key: The paper strongly demonstrates that a massive amount of unlabeled crystal data can be effectively used for pre-training, significantly reducing the reliance on expensive, labeled datasets for downstream tasks.
  • Distillation Outperforms Fine-tuning: The knowledge distillation approach is shown to be more effective than traditional pre-train-and-fine-tune methods. This suggests that directly transferring learned representations and relationships is more beneficial than simply initializing a model and then retraining it.
  • Combined Node and Graph-Level Pre-training is Crucial: Ablation studies reveal that incorporating both local (node-level) and global (graph-level) structural information during pre-training leads to the most significant improvements in downstream property prediction.
  • Mitigating DFT Bias: Distilled knowledge from CrysGNN helps property predictors trained on experimental data to reduce the inherent bias introduced by DFT calculations, leading to more accurate predictions closer to experimental reality.
  • Generalizability of the Approach: The distillation framework is shown to be effective across various SOTA GNN architectures, highlighting its broad applicability.

Key Results

  • Significant Performance Gains: Distilled versions of SOTA models (CGCNN, CrysXPP, GATGNN, ALIGNN) consistently outperformed their vanilla versions across multiple properties. Average improvements ranged from 4.19% to 16.20%.
  • Robustness on Sparse Data: The distilled models showed even greater improvement when trained on limited data splits (e.g., 20% of the training data), demonstrating the robustness of the pre-training framework.
  • Comparison with Existing Pre-trained Models: Distilled CGCNN outperformed finetuned CrysGNN and other baselines like Pretrain-GNN on the JARVIS dataset, with Pretrain-GNN performing the worst due to its design for simpler molecular structures.
  • Removal of DFT Bias: When trained with a small percentage of experimental data, distilled models showed a consistently larger degree of improvement in reducing MAE for formation energy compared to their vanilla counterparts.

Limitations and Future Work

  • Model Complexity: The authors note that for more complex models like ALIGNN and GATGNN, the improvement from distillation was less pronounced compared to simpler models like CGCNN. They suggest that designing deeper pre-training models or incorporating additional information (like angle-based features or attention mechanisms) might further enhance performance.
  • Future Work: The authors propose extending the pre-training framework to incorporate multimodal information, such as text descriptions and images of crystals, in addition to structural graph information. This could lead to even richer representations and improved prediction capabilities.

Summary updated

Abstract

In recent years, graph neural network (GNN) based approaches have emerged as a powerful technique to encode complex topological structure of crystal materials in an enriched repre- sentation space. These models are often supervised in nature and using the property-specific training data, learn relation- ship between crystal structure and different properties like formation energy, bandgap, bulk modulus, etc. Most of these methods require a huge amount of property-tagged data to train the system which may not be available for different prop- erties. However, there is an availability of a huge amount of crystal data with its chemical composition and structural bonds. To leverage these untapped data, this paper presents CrysGNN, a new pre-trained GNN framework for crystalline materials, which captures both node and graph level structural information of crystal graphs using a huge amount of unla- belled material data. Further, we extract distilled knowledge from CrysGNN and inject into different state of the art prop- erty predictors to enhance their property prediction accuracy. We conduct extensive experiments to show that with distilled knowledge from the pre-trained model, all the SOTA algo- rithms are able to outperform their own vanilla version with good margins. We also observe that the distillation process provides significant improvement over the conventional ap- proach of finetuning the pre-trained model. We will release the pre-trained model along with the large dataset of 800K crys- tal graph which we carefully curated; so that the pre-trained model can be plugged into any existing and upcoming models to enhance their prediction accuracy.
BibTeX
@inproceedings{das2023crysgnn,
  author = {Kishalay Das and Bidisha Samanta and Pawan Goyal and Seung-Cheol Lee and Satadeep Bhattacharjee and Niloy Ganguly},
  title = {{CrysGNN: Distilling Pre-trained Knowledge to Enhance Property Prediction for Crystalline Materials}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  url = {https://openreview.net/forum?id=jqhyzgv7gD},
}
Figure