Quantitative Analysis of Metallographic Image Using Attention-Aware Deep Neural Networks

Yifei Xu, Yuewan Zhang, Meizi Zhang, Mian Wang, Wujiang Xu, Chaoyong Wang, Yan Sun, Pingping WeiSensors 2021 · OpenReview

TL;DR

This paper introduces two novel deep learning models, MAUNet and SASAPD, for the quantitative analysis of metallographic images. These models leverage attention mechanisms to improve the accuracy of identifying microstructural components in both single-phase and multi-phase steel images, outperforming existing state-of-the-art methods.

What Problem Does This Paper Address?

Metallographic analysis is crucial for understanding metal quality and predicting mechanical properties. Traditionally, this involves manual identification and characterization of microstructures in images. While deep learning has shown promise, accurately and efficiently detecting and segmenting these complex structures, especially in multi-phase materials with varying element sizes, remains a significant challenge. Existing methods often struggle with issues like lost high-frequency details, irrelevant feature information, and difficulties in handling imbalanced datasets and objects of different scales.

Methods

The paper proposes two distinct deep learning architectures:

  • Modified Attention U-Net (MAUNet): Designed for single-phase metallographic images, MAUNet builds upon the U-Net architecture. It introduces a "frequency-aware encoder" (FAE) to better preserve high-frequency information lost in traditional U-Net skip connections. Additionally, a "dual-path attention module" is incorporated into the decoder to enhance feature interpretability by combining spatial and channel attention.
  • Self-adaptive Attention-aware Soft Anchor-Point Detector (SASAPD): For multi-phase metallographic images, SASAPD is an anchor-free object detection model. It enhances the existing SAPD framework with several key improvements:
  • Sampling Reweighting Strategy (SRS): This strategy aims to prevent "attention bias" by re-ranking and re-weighting anchor points, giving more importance to positive samples and improving detection of smaller objects.
  • Soft Self-adaptive Selection (3S) strategy: This approach allows the model to adaptively select the most informative pyramid levels for each instance, rather than relying on fixed heuristics or one-hot encodings.
  • Attention-aware Lightweight Feature Pyramid (ALFP): This module integrates attention mechanisms into the feature pyramid to better capture context between adjacent feature maps and improve the perception of fine-grained features.

Datasets

The authors introduce two new metallographic image datasets:

  • Single-Phase Metallographic Image Dataset (SPMID): This dataset comprises 7500 training, 500 validation, and 1433 testing images of hot-rolled section steel (20G). It is categorized by carbon level and segregation degree, used for evaluating segmentation performance.
  • Multi-Phase Metallographic Image Dataset (MPMID): This dataset consists of 6500 training, 500 validation, and 1000 testing images of the longitudinal section of round steel (42CrMo). It features structures like upper bainite (up B), ferrite (F), pearlite (P), and segregation bands (Segband), with significant variations in object sizes. This dataset is used for object detection tasks.

Key Insights and Ideas

  • High-frequency information is critical for metallographic segmentation: The MAUNet's FAE module demonstrates that explicitly preserving high-frequency details in skip connections significantly improves segmentation accuracy, especially for fine structures like carbides.
  • Attention mechanisms enhance feature interpretability and discriminability: Both MAUNet's dual-path attention and SASAPD's attention blocks help the models focus on relevant regions and features, leading to more accurate detection and segmentation.
  • Adaptive feature selection is key for multi-phase object detection: SASAPD's 3S strategy and ALFP module show that dynamically assigning contributions from different feature pyramid levels based on instance characteristics is more effective than fixed approaches for handling objects of varying sizes.
  • Addressing attention bias is crucial for small object detection: The SRS strategy in SASAPD helps mitigate the tendency of models to focus on larger or more prominent objects, improving the detection of smaller microstructural constituents.
  • Careful preprocessing is vital: The paper highlights the importance of a multi-step preprocessing pipeline (grayscale transformation, noise reduction, sharpening, binarization, and morphological processing) for improving image quality and subsequent model performance.

Key Results

  • MAUNet: Achieved state-of-the-art performance on the SPMID dataset for segmentation, with a Dice score of 0.963 and IoU of 0.923, outperforming other comparative methods like mU-Net and Deeplab V3+. It also demonstrated comparable parameters and running time to baseline U-Net.
  • SASAPD: Showed superior performance on the MPMID dataset for object detection. It achieved the highest Dice scores for both ferrite (F) and pearlite (P) (0.963 and 0.947, respectively) and improved Precision and Recall metrics compared to methods like SAPD, AutoAssign, and YoloV4. The model also maintained competitive inference speeds (FPS).

Limitations and Future Work

  • Determining optimal network stages and pyramid levels: The current number of stages and pyramid levels are determined empirically. The authors suggest exploring Network Architecture Search (NAS) to automate this design process.
  • Initialization of pyramid levels: The performance of SASAPD relies on the initial assignment of instances to pyramid levels. The authors aim to develop methods to eliminate this dependency on initialization in future work.
  • Generalizability of preprocessing: While the preprocessing pipeline significantly improved image quality, its effectiveness might vary across different types of metallographic images.
  • Computational cost of attention mechanisms: While the proposed models are efficient, attention mechanisms can still add computational overhead, which might be a consideration for real-time applications on resource-constrained devices.

Summary updated

Abstract

As a detection tool to identify metal or alloy, metallographic quantitative analysis has received increasing attention for its ability to evaluate quality control and reveal mechanical properties. The detection procedure is mainly operated manually to locate and characterize the constitution in metallographic images. The automatic detection is still a challenge even with the emergence of several excellent models. Benefiting from the development of deep learning, with regard to two different metallurgical structural steel image datasets, we propose two attention-aware deep neural networks, Modified Attention U-Net (MAUNet) and Self-adaptive Attention-aware Soft Anchor-Point Detector (SASAPD), to identify structures and evaluate their performance. Specifically, in the case of analyzing single-phase metallographic image, MAUNet investigates the difference between low-frequency and high-frequency and prevents duplication of low-resolution information in skip connection used in an U-Net like structure, and incorporates spatial-channel attention module with the decoder to enhance interpretability of features. In the case of analyzing multi-phase metallographic image, SASAPD explores and ranks the importance of anchor points, forming soft-weighted samples in subsequent loss design, and self-adaptively evaluates the contributions of attention-aware pyramid features to assist in detecting elements in different sizes. Extensive experiments on the above two datasets demonstrate the superiority and effectiveness of our two deep neural networks compared to state-of-the-art models on different metrics.
BibTeX
@inproceedings{xu2021quantitative,
  author = {Yifei Xu and Yuewan Zhang and Meizi Zhang and Mian Wang and Wujiang Xu and Chaoyong Wang and Yan Sun and Pingping Wei},
  title = {{Quantitative Analysis of Metallographic Image Using Attention-Aware Deep Neural Networks}},
  booktitle = {Sensors},
  year = {2021},
  url = {https://openreview.net/forum?id=0_XhF7T_LY},
}

Loading related content...

Figure