Microstructural segmentation using a union of attention guided U-Net models with different color transformed images

Momojit Biswas, Rishav Pramanik, Shibaprasad Sen, Aleksandr Sinitca, Dmitrii Kaplun, Ram SarkarScientific 2023 · DOI

TL;DR

This paper introduces an ensemble of attention-guided U-Net models for segmenting metallographic images, leveraging different color spaces (RGB, HSV, YUV) to capture complementary microstructural information. The proposed method achieves competitive results with fewer parameters than state-of-the-art approaches, offering a more efficient solution for material characterization.

What Problem Does This Paper Address?

Understanding the microstructure of metals is crucial for predicting their mechanical properties, performance, and potential failure modes in various engineering applications. Traditionally, this analysis is done manually, which is time-consuming and labor-intensive. While image processing and deep learning have emerged as powerful tools for automating this task, accurately segmenting complex microstructures, especially those with subtle features and class imbalances, remains a challenge. Existing deep learning methods often require significant computational resources and may not fully exploit the rich information present in different representations of the same image.

Methods

The core of this work is an ensemble approach that combines the strengths of multiple U-Net models. Specifically, the authors propose:

  • Color Space Transformation: The original RGB metallographic images are transformed into two additional color spaces: HSV (Hue, Saturation, Value) and YUV (Luminance, Chrominance). The rationale is that these different color spaces capture distinct aspects of the microstructure, offering complementary information.
  • Attention-Guided Dilated U-Net: Each color space is fed into a modified U-Net architecture. This U-Net incorporates:
  • Dilated Convolutions: These are used to increase the receptive field of the convolutional layers without increasing the number of parameters or computational cost, allowing the model to capture multi-scale global information and reduce overfitting.
  • Attention Modules: Both channel and spatial attention mechanisms are integrated into the U-Net. These modules help the network focus on the most discriminative features within the image, improving the accuracy of segmentation, particularly for fine-grained details.
  • Ensemble Strategy: The segmentation masks generated by the three individual U-Net models (one for each color space) are combined using a sum-rule-based ensemble method to produce the final prediction mask. This aggregation aims to leverage the complementary information learned by each model.

Datasets

The proposed method is evaluated on the MetalDAM dataset. This is a publicly available benchmark dataset specifically designed for metallographic image segmentation, containing 42 labeled images with resolutions of 1280x895 and 1024x703. The images are annotated pixel-wise for five micro-constituent classes: Matrix, Martensite/Austenite (MA), Precipitates, Austenite, and Defects.

Key Insights and Ideas

  • Complementary Information in Color Spaces: The study strongly suggests that different color spaces (RGB, HSV, YUV) provide distinct and valuable information about microstructures. Ensembling models trained on these different representations is more effective than using any single color space alone.
  • Attention Enhances Feature Discrimination: The integration of channel and spatial attention modules significantly improves the U-Net's ability to learn fine-grained features and highlight discriminative regions, leading to more accurate segmentation.
  • Dilated Convolutions for Efficiency: Using dilated convolutions allows the U-Net to capture broader contextual information without a proportional increase in model complexity or parameters, making it more efficient.
  • Ensembling for Robustness: The ensemble approach effectively mitigates the weaknesses of individual models, leading to more robust and accurate segmentation results, especially in the presence of class imbalance.

Key Results

The proposed ensemble method achieved a mean Intersection over Union (IoU) score of 0.677 on the MetalDAM dataset. This performance is comparable to state-of-the-art methods, but with a significantly fewer number of model parameters (597,637 for individual RGB U-Net vs. over 6 million for some baseline U-Net models). The authors also demonstrated that their method outperforms individual U-Net models trained on single color spaces. A statistical test (Wilcoxon Rank-Sum) confirmed the statistical significance of the ensemble approach over individual base models.

Limitations and Future Work

  • Dataset Size: The authors acknowledge that the MetalDAM dataset is relatively small, which can lead to some instability in training and potential overfitting. They suggest that larger datasets would further improve performance.
  • Training Time: Training three separate U-Net models increases the overall training time compared to a single model.
  • Boundary Overestimation: The predicted masks sometimes overestimate boundary regions, which might be related to the scarcity of boundary pixels in the dataset.

Future work suggested by the authors includes:

  • Investigating boundary attention mechanisms to improve the accuracy of edge detection.
  • Exploring region proposal-based architectures (like Mask R-CNN) to handle multiple chrominance transformations within a single architecture, potentially reducing training time.
  • Leveraging self-supervised learning (SSL) techniques to pre-train networks using both labeled and unlabeled data, especially given the limited size of available labeled datasets.
  • Developing and testing on larger, more diverse benchmark datasets for metallographic image segmentation.

Summary updated

Abstract

Metallographic images or often called the microstructures contain important information about metals, such as strength, toughness, ductility, corrosion resistance, which are used to choose the proper materials for various engineering applications. Thus by understanding the microstructures, one can determine the behaviour of a component made of a particular metal, and can predict the failure of that component in certain conditions. Image segmentation is a powerful technique for determination of morphological features of the microstructure like volume fraction, inclusion morphology, void, and crystal orientations. These are some key factors for determining the physical properties of metal. Therefore, automatic micro-structure characterization using image processing is useful for industrial applications which currently adopts deep learning-based segmentation models. In this paper, we propose a metallographic image segmentation method using an ensemble of modified U-Nets. Three U-Net models having the same architecture are separately fed with color transformed imaged (RGB, HSV and YUV). We improvise the U-Net with dilated convolutions and attention mechanisms to get finer grained features. Then we apply the sum-rule-based ensemble method on the outcomes of U-Net models to get the final prediction mask. We achieve the mean intersection over union (IoU) score of 0.677 on a publicly available standard dataset, namely MetalDAM. We also show that the proposed method obtains results comparable to state-of-the-art methods with fewer number of model parameters. The source code of the proposed work can be found at https://github.com/mb16biswas/attention-unet .
BibTeX
@inproceedings{biswas2023microstructural,
  author = {Momojit Biswas and Rishav Pramanik and Shibaprasad Sen and Aleksandr Sinitca and Dmitrii Kaplun and Ram Sarkar},
  title = {{Microstructural segmentation using a union of attention guided U-Net models with different color transformed images}},
  booktitle = {Scientific},
  year = {2023},
  doi = {10.1038/s41598-023-32318-9},
}
Figure