Mixed-UNet: Refined class activation mapping for weakly-supervised semantic segmentation with multi-scale inference

Yang Liu, Lijin Lian, Ersi Zhang, Lulu Xu, Chufan Xiao, Xiaoyun Zhong, Fang Li, Bin Jiang, Yuhan Dong, Lan Ma, Qiming Huang, Ming Xu, Yongbing Zhang, Dongmei Yu, Chenggang Yan, Peiwu Qin

TL;DR

This paper tackles the challenge of accurately segmenting lesions in medical images (like brain MRI) using only image-level labels, which are much easier to obtain than pixel-level annotations. They propose a novel "Mixed-UNet" architecture that refines initial lesion localization maps (CAMs) by considering multiple image scales and employing a dual-branch decoder, leading to improved segmentation accuracy.

What Problem Does This Paper Address?

Accurate segmentation of lesions in medical images is crucial for diagnosis and treatment planning. Deep learning models excel at this, but they typically require extensive pixel-level annotations, which are time-consuming and expensive to create. Weakly-supervised semantic segmentation (WSSS) aims to overcome this by using less precise labels, such as image-level tags (e.g., "lesion present"). A common WSSS approach uses Class Activation Maps (CAMs) to highlight regions of interest. However, standard CAMs often produce coarse localization maps that struggle to capture the precise boundaries of lesions, especially for small or scattered ones, limiting their effectiveness for segmentation. This paper addresses this gap by proposing a method to refine CAMs and a new network architecture to leverage these refined maps for more accurate lesion segmentation.

Methods

The core of the proposed solution involves two main components:

  1. Multi-scale Inference for CAM Refinement: The authors acknowledge that single-scale CAMs can miss details. To address this, they introduce a multi-scale inference strategy. This involves generating CAMs at different image scales (by resizing the input image) and then fusing these multi-scale CAMs. The idea is that different scales capture complementary information, leading to more robust and detailed lesion localization. This refined CAM is then used to generate "pseudo-masks" which serve as pixel-level supervision for the segmentation network. They also mention using a dense Conditional Random Field (CRF) to further optimize these pseudo-masks, especially for boundary refinement.
  2. Mixed-UNet Architecture: For the actual segmentation, they propose a novel network called "Mixed-UNet," inspired by the U-Net architecture. The key innovation lies in its decoding phase, which features two parallel branches. This dual-branch structure allows the network to extract and combine features from different perspectives, aiming to capture more subtle details and improve segmentation accuracy compared to a single-branch decoder. The encoding phase is shared, and the outputs from the two decoding branches are fused before the final segmentation output. This design is intended to improve performance without a significant increase in parameters or computational cost.

Datasets

The paper utilizes two datasets:

  • ACI Dataset: This is a custom dataset collected from the South University of Science and Technology Hospital. It consists of MRI scans from patients with acute cerebral infarction (ACI) and normal brain MRI slices. The dataset is split into training and testing sets.
  • ICH Validation Dataset: This is a publicly available dataset from Kaggle containing CT images of intracerebral hemorrhage (ICH). It's used to validate the model's robustness and generalizability.

Key Insights and Ideas

  • Multi-scale CAMs are better than single-scale: The intuition that different image scales reveal different aspects of an object is applied to CAM generation. Fusing CAMs from multiple scales significantly improves the precision of lesion localization compared to using a single scale.
  • Dual-branch decoder for richer feature extraction: The Mixed-UNet's parallel decoding branches are designed to capture more nuanced information, which is crucial for segmenting small or complex lesions where single-branch decoders might struggle.
  • Weak supervision can be powerful with refinement: The paper demonstrates that even with simple image-level labels, sophisticated refinement techniques (multi-scale CAMs, CRF) can generate effective pseudo-masks that enable strong segmentation performance.
  • Trade-off between accuracy and speed: While the Mixed-UNet achieves high accuracy, its parallel branch structure leads to a longer inference time compared to some other models, highlighting a common trade-off in deep learning architectures.

Key Results

  • Improved Segmentation Accuracy: The Mixed-UNet, when trained with refined CAMs, outperforms other WSSS methods on their ACI dataset. For the ICH dataset, the CRF-optimized pseudo-masks achieve high PA (0.9831), MIoU (0.7706), and Dice (0.6907).
  • Quantitative Comparisons: Table III shows that Mixed-UNet achieves competitive results in terms of PA and MIoU compared to FCN, U-Net, and DeepLab V3, with a notable improvement in Dice score (0.560 vs. 0.529 for DeepLab V3). When combined with CRF, the MIoU further increases.
  • Efficiency: Mixed-UNet has a significantly lower number of trainable parameters (10.3M) and memory consumption (50M) compared to DeepLab V3 (39.6M parameters, 155M memory), making it more resource-efficient in terms of model size.
  • Qualitative Results: Figures 4, 5, 6, and 7 visually demonstrate the effectiveness of multi-scale CAMs in refining lesion boundaries and the superior segmentation results of Mixed-UNet compared to other methods, especially in capturing lesion details.

Limitations and Future Work

  • Label Dependence: The authors acknowledge that their method, like most WSSS approaches, is dependent on the accuracy of the initial labels. Investigating how to handle inaccurate labels further is a direction for future work.
  • Inference Time: The parallel branch structure of Mixed-UNet, while beneficial for accuracy, increases inference time. Further optimization might be needed for real-time applications.
  • Generalizability: While validated on a public ICH dataset, extending the model to other imaging modalities and lesion types is suggested for future research. The authors believe the model's architecture is flexible enough for such extensions.

Summary updated

Abstract

Deep learning techniques have shown great potential in medical image processing, particularly through accurate and reliable image segmentation on magnetic resonance imaging (MRI) scans or computed tomography (CT) scans, which allow the localization and diagnosis of lesions. However, training these segmentation models requires a large number of manually annotated pixel-level labels, which are time-consuming and labor-intensive, in contrast to image-level labels that are easier to obtain. It is imperative to resolve this problem through weakly-supervised semantic segmentation models using image-level labels as supervision since it can significantly reduce human annotation efforts. Most of the advanced solutions exploit class activation mapping (CAM). However, the original CAMs rarely capture the precise boundaries of lesions. In this study, we propose the strategy of multi-scale inference to refine CAMs by reducing the detail loss in single-scale reasoning. For segmentation, we develop a novel model named Mixed-UNet, which has two parallel branches in the decoding phase. The results can be obtained after fusing the extracted features from two branches. We evaluate the designed Mixed-UNet against several prevalent deep learning-based segmentation approaches on our dataset collected from the local hospital and public datasets. The validation results demonstrate that our model surpasses available methods under the same supervision level in the segmentation of various lesions fr...
BibTeX
@article{liu2022mixed,
  author = {Yang Liu and Lijin Lian and Ersi Zhang and Lulu Xu and Chufan Xiao and Xiaoyun Zhong and Fang Li and Bin Jiang and Yuhan Dong and Lan Ma and Qiming Huang and Ming Xu and Yongbing Zhang and Dongmei Yu and Chenggang Yan and Peiwu Qin},
  title = {{Mixed-UNet: Refined class activation mapping for weakly-supervised semantic segmentation with multi-scale inference}},
  journal = {Frontiers of Computer Science},
  year = {2022},
  url = {https://openreview.net/forum?id=HnqYX4fvfF},
}

Loading related content...

Figure