Semi-supervised teacher-student deep neural network for materials discovery

Daniel Gleaves, Edirisuriya M. Dilanga Siriwardane, Yong Zhao, Nihang Fu, Jianjun HuarXiv 2021 · arXiv

TL;DR

This paper introduces a novel semi-supervised deep learning framework, TSDNN, to tackle the challenge of discovering new materials. By leveraging both limited labeled data and abundant unlabeled data, TSDNN significantly improves the accuracy of predicting material stability (formation energy) and synthesizability, outperforming existing methods and enabling more efficient large-scale materials discovery.

What Problem Does This Paper Address?

Discovering new materials with desired properties is crucial for technological advancement. While generative models can create millions of hypothetical material candidates, efficiently filtering these candidates for stability and synthesizability remains a major bottleneck. Existing machine learning models struggle due to a significant lack of "negative" samples (unstable or non-synthesizable materials) in public databases, leading to biased models that perform poorly on out-of-distribution data. This paper addresses this critical data scarcity and bias problem by proposing a semi-supervised learning approach.

Methods

The core of this work is the Teacher-Student Deep Neural Network (TSDNN) framework. TSDNN employs a dual-network architecture where a "teacher" model generates pseudo-labels for unlabeled data, which are then used to train a "student" model. This process is iterative, with the student's performance providing feedback to refine the teacher's pseudo-labeling, creating a dynamic and adaptive learning system. This approach effectively leverages the vast amount of unlabeled data to overcome the limitations of scarce labeled data and mitigate biases.

For material property prediction, the authors utilize the Crystal Graph Convolutional Neural Network (CGCNN) architecture, which represents materials as graphs to capture structural information. TSDNN is integrated with CGCNN for both formation energy classification and synthesizability prediction.

Datasets

  • Materials Project (MP) database: Used as the primary source for labeled materials, including both known stable materials (negative formation energy) and synthesizable materials.
  • ICSD database: Also used as a source for known materials.
  • CubicGAN-generated materials: Millions of hypothetical cubic crystal structures generated by the authors' CubicGAN model served as a large source of unlabeled data for training and as a benchmark test set for evaluating the models' performance on novel, out-of-distribution materials.

A key aspect of their methodology is the careful preparation of training and testing datasets, including strategies for selecting "negative" samples from unlabeled data to create more balanced and representative datasets for training.

Key Insights and Ideas

  • Data Bias is a Major Hurdle: The paper strongly emphasizes that the inherent bias in materials databases (over-representation of stable/synthesizable materials) is a fundamental challenge for traditional supervised learning in materials discovery.
  • Semi-Supervised Learning is Key: TSDNN's success highlights the power of semi-supervised learning to overcome data scarcity and bias by effectively utilizing abundant unlabeled data.
  • Teacher-Student Dynamics: The iterative feedback loop between the teacher and student models is crucial for refining pseudo-labels and improving the student's ability to generalize to unseen data.
  • Evaluation Matters: The authors stress the importance of using appropriate evaluation strategies, particularly the inclusion of a negative class in test sets, to avoid artificially inflated performance metrics and to truly assess a model's ability to differentiate between stable/synthesizable and unstable/non-synthesizable materials.
  • Generative Models + Screening = Discovery: The paper demonstrates a powerful synergy between generative models (CubicGAN) for creating novel candidates and advanced screening models (TSDNN) for identifying promising ones.

Key Results

  • Synthesizability Prediction: TSDNN achieved a 97.90% true positive rate (TPR) on a test set of materials structurally similar to the training data, significantly outperforming the baseline PU-learning method (87.90% TPR). On a more challenging test set with structurally different materials, TSDNN achieved a 94.11% accuracy and 92.90% TPR.
  • Formation Energy Prediction: The Unseparated TSDNN model achieved a 74.60% F1 score and 74% accuracy for classifying materials with negative formation energies, representing an absolute 10.3% improvement in F1 score and 9.7% improvement in accuracy compared to the baseline CGCNN regression model. The Separated TSDNN model achieved 100% precision, indicating a highly reliable identification of stable materials.
  • New Materials Discovery: When applied to 2.5 million CubicGAN-generated cubic structures, TSDNN identified 1000 candidate materials. DFT validation confirmed that 51.2% of these top candidates have negative formation energies, demonstrating the framework's effectiveness in discovering novel, stable materials.

Limitations and Future Work

  • Dataset Similarity Assumption: The TSDNN framework relies on the assumption that the unlabeled dataset contains samples similar to the labeled dataset, which is crucial for effective pseudo-labeling.
  • Label Accuracy: The performance of the student model is sensitive to the accuracy of the initial labeled dataset. While the feedback mechanism can resolve some teacher inaccuracy, a fundamentally flawed labeled dataset would hinder performance.
  • Computational Cost: While TSDNN uses simpler models than some previous approaches, training deep neural networks can still be computationally intensive.
  • Future Work: The authors suggest applying their TSDNN framework to other materials property prediction tasks and exploring its integration with other generative models for broader materials discovery applications. They also highlight the potential for further optimization of dataset configurations and model architectures.

Summary updated

BibTeX
@misc{gleaves2021semi,
  author = {Daniel Gleaves and Edirisuriya M. Dilanga Siriwardane and Yong Zhao and Nihang Fu and Jianjun Hu},
  title = {{Semi-supervised teacher-student deep neural network for materials discovery}},
  year = {2021},
  eprint = {2112.06142},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2112.06142},
}
Figure