Learning-based defect recognition for quasi-periodic HRSTEM images

Nik Dennler, Antonio Foncubierta–Rodríguez, Titus Neupert, Marilyne Sousa

TL;DR

This paper introduces a novel semi-supervised machine learning approach to automate the detection of crystalline defects in high-resolution transmission electron microscopy (HRSTEM) images. By combining a convolutional neural network for initial classification with custom-generated filters, the method efficiently highlights symmetry-breaking defects and amorphous regions, significantly reducing the manual effort and ambiguity in material analysis.

What Problem Does This Paper Address?

Analyzing crystalline material defects is critical for understanding and controlling material properties, especially in advanced semiconductor devices. High-resolution scanning transmission electron microscopy (HRSTEM) provides the necessary atomic-scale resolution to visualize these defects. However, current defect identification relies heavily on human expertise, which is a time-consuming, tedious, and often subjective process. This can lead to inconsistent results and limits the accessibility of this powerful imaging technique to non-experts. The paper aims to bridge this gap by developing an automated system that assists in defect recognition, making the process more robust, reproducible, and efficient.

Methods

The proposed method employs a multi-stage pipeline:

  1. Semi-supervised Learning for Patch Classification: A Convolutional Neural Network (CNN), specifically a VGG16 architecture adapted for this task, is trained to classify image patches as either "defective" or "non-defective." To address the challenge of limited labeled data, the authors utilize transfer learning from ImageNet and extensive data augmentation techniques.
  2. Best-Patch Heuristic: A graph-based heuristic is used to select an optimal "model" patch from the set of non-defective patches identified by the CNN. This model patch represents the ideal crystalline structure and is crucial for generating the subsequent filters.
  3. Automatic Filter Generation:
    • Symmetry Filtering: An "Eigenfilter" technique is adapted to generate convolutional filters that specifically highlight symmetry-breaking defects like stacking faults, twin defects, and grain boundaries. These filters are derived from the chosen non-defective model patch.
  4. Variance Filtering: A separate variance filter is applied to segment regions of low local variance, which typically correspond to amorphous regions or beam defects.
  5. Defect Segmentation: The generated filters are convolved with the preprocessed HRSTEM images to produce defect maps. The Mahalanobis distance is used to quantify defectiveness for symmetry filters, while thresholding is applied to the variance map.

Datasets

The study uses a dataset of HRSTEM images of III-V/Si crystalline materials.

  • Training Data: 10 defect-containing images were used for training the CNN classifier, with patches augmented to create a balanced dataset of approximately 2,500 patches per class (defective/non-defective).
  • Testing Data: 3 defect-containing images and 28 non-defective images were used for evaluation. These were cropped into non-overlapping patches without augmentation.

Key Insights and Ideas

  • Leveraging "Good" Examples: The core idea of using a "best-patch" heuristic to define a clean reference structure for filter generation is a clever way to adapt filter design to the specific material and imaging conditions, rather than relying on generic filters.
  • Combining Data-Driven and Model-Based Approaches: The synergy between the data-driven CNN for initial classification and the model-based, learned filters for highlighting specific defect types is a powerful combination. This allows the system to benefit from the generality of deep learning while retaining the precision of image filtering for structural anomalies.
  • Robustness to Small Datasets: The successful application of transfer learning and data augmentation demonstrates that effective defect recognition can be achieved even with limited labeled HRSTEM data, which is a common challenge in materials science.
  • Addressing Scale Invariance: The paper implicitly addresses the scale-invariance problem of defect detection by using a CNN that can learn features at different scales and by generating filters that are sensitive to local symmetry breaking.

Key Results

  • The CNN classifier achieved good performance, with accuracy around 92.43% on the training set and 88.06% on the test set.
  • The system demonstrated strong performance on unseen non-defective images, with a specificity of 99.53%.
  • Expert evaluation of the filtering results indicated that the assistance was "Very helpful" in 69.2% of cases, "Potentially helpful" in 23.1%, and "Inadequate" in only 7.7%.
  • The computational time for patch prediction and filtering is relatively fast, making it practical for routine analysis.

Limitations and Future Work

  • Ambiguous Patches: The paper acknowledges that patches with 1%-10% relative defective area were excluded from training due to ambiguity. While these are included in evaluation, their classification might still be challenging.
  • Generalizability to Other Materials: While tested on III-V/Si, the authors suggest the workflow could be applied to other atomic imaging techniques and materials, implying further validation is needed.
  • Future Work: The authors propose extending the pipeline to classify broader types of crystallographic defects and identify novel, yet unidentified, crystallographic arrangements. They also mention that the entire code is available as an open-source tool.

Summary updated

Abstract

Controlling crystalline material defects is crucial, as they affect properties of the material that may be detrimental or beneficial for the final performance of a device. Defect analysis on the sub-nanometer scale is enabled by high-resolution scanning transmission electron microscopy (HRSTEM), where the identification of defects is currently carried out based on human expertise. However, the process is tedious, highly time consuming and, in some cases, yields ambiguous results. Here we propose a semi-supervised machine learning method that assists in the detection of lattice defects from atomic resolution HRSTEM images. It involves a convolutional neural network that classifies image patches as defective or non-defective, a graph-based heuristic that chooses one non-defective patch as a model, and finally an automatically generated convolutional filter bank, which highlights symmetry breaking such as stacking faults, twin defects and grain boundaries. Additionally, we suggest a variance filter to segment amorphous regions and beam defects. The algorithm is tested on III-V/Si crystalline materials and successfully evaluated against different metrics and a baseline approach, showing promising results even for extremely small training data sets and for noise compromised images. By combining the data-driven classification generality, robustness and speed of deep learning with the effectiveness of image filters in segmenting faulty symmetry arrangements, we provide a valuable open-source tool to the microscopist community that can streamline future HRSTEM analyses of crystalline materials.
BibTeX
@inproceedings{dennler2021learning,
  author = {Nik Dennler and Antonio Foncubierta–Rodríguez and Titus Neupert and Marilyne Sousa},
  title = {{Learning-based defect recognition for quasi-periodic HRSTEM images}},
  booktitle = {Micron},
  year = {2021},
  doi = {10.1016/j.micron.2021.103069},
}
Figure