Cut-Paste Consistency Learning for Semi-Supervised Lesion Segmentation

Boon Peng Yap, Beng Koon NgWACV 2023

TL;DR

This paper introduces "Cut-Paste Consistency Learning," a novel semi-supervised method for lesion segmentation that leverages cut-paste augmentation to create synthetic partially labeled data. By combining a standard supervised loss with a new background consistency loss, the method achieves strong performance on medical imaging tasks with limited labeled data, outperforming existing semi-supervised techniques without complex architectural changes.

What Problem Does This Paper Address?

Medical image analysis, particularly lesion segmentation, is severely hampered by the scarcity of labeled data. Acquiring these labels is expensive and time-consuming, requiring specialized expertise. While semi-supervised learning offers a solution by utilizing unlabeled data, existing methods often rely on complex architectural modifications or multiple training rounds. This paper aims to develop a simpler, more data-efficient semi-supervised approach for lesion segmentation that can achieve competitive results with fewer computational overheads.

Methods

The core of the proposed method lies in its innovative use of cut-paste augmentation to generate synthetic training data. Here's how it works:

  • Synthetic Data Generation:
    • Foreground objects (lesions) are extracted from labeled images using their segmentation masks.
  • These extracted objects are then "pasted" onto unlabeled images.
  • To ensure visual realism, a color matching strategy is employed. For color images, this involves matching based on Delta E distances in the CIELAB color space. For grayscale images (like CT scans), it uses L2 distance in pixel intensity. This is a key improvement over random pasting, which can lead to visually inconsistent and thus less useful synthetic samples.
  • Image Blending involves adding geometric (rotation, translation, resizing) and color jittering to the pasted objects, followed by Gaussian blurring to blend them seamlessly. A post-processing step removes any out-of-bound pasted objects.
  • Training Objective:
    • Supervised Loss: The standard binary cross-entropy loss is applied to the generated partially labeled synthetic images.
  • Background Consistency Loss: Since the regions outside the pasted lesions in synthetic images are still considered unlabeled, a new background consistency term is introduced. This loss encourages the network's predictions for these background regions in the synthetic image to be consistent with the predictions for the same regions in the original unlabeled image. This is achieved by using a target network (either a copy or an EMA version of the main network) and minimizing the mean squared error between their background predictions.

This approach allows the model to learn from both labeled and synthetically augmented unlabeled data in a single training pass, avoiding the need for iterative pseudo-labeling.

Datasets

The proposed method was evaluated on two publicly available datasets:

  • Indian Diabetic Retinopathy Image Dataset (IDRiD): This dataset contains color fundus images with annotations for four types of retinal lesions: microaneurysms, hemorrhage, hard exudates, and soft exudates. For semi-supervised learning, 435 additional images were treated as unlabeled.
  • CT-ICH Dataset: This dataset consists of CT scans for intracranial hemorrhage segmentation. It's noted as being heavily imbalanced, with many slices not containing hemorrhages. The dataset was split into labeled and unlabeled portions using patient-wise stratified sampling.

No new benchmarks were introduced; the focus was on applying the method to existing ones.

Key Insights and Ideas

  • Cut-Paste for Data Augmentation is Powerful: The paper demonstrates that strategically cutting and pasting foreground objects from labeled data onto unlabeled backgrounds can create highly effective synthetic training data for semi-supervised learning, especially for small, sparse objects like lesions.
  • Visual Realism Matters: Simply pasting objects randomly leads to poor results. The introduction of color matching (using Delta E or L2 distance) to align the pasted object's appearance with the background significantly improves the realism and utility of the synthetic images.
  • Background Consistency is Crucial: Forcing consistency in the background regions of synthetic images (where labels are uncertain) is vital. This prevents the model from being penalized for false negatives in these areas and helps it learn more robust representations.
  • Simplicity is Key: The method avoids complex architectural changes, making it easy to integrate into existing segmentation frameworks and computationally efficient, requiring only one training round.

Key Results

  • IDRiD Dataset: The proposed method consistently ranked among the top performers, achieving the best performance in hemorrhage segmentation. When extended with the Mean Teacher model, it achieved the best average result across all four lesion types, showing a significant improvement over the fully supervised baseline and other semi-supervised methods.
  • CT-ICH Dataset: On this imbalanced dataset, the cut-paste consistency method significantly outperformed the supervised baseline, especially in low-label settings (e.g., 30% labeled data). It emerged as the best-performing method in terms of Jaccard indices across all semi-supervised settings.
  • Ablation Studies: The studies confirmed the importance of each component. The background consistency term provided the largest relative performance gain (3.23% improvement). Color matching also showed a notable positive impact.

Limitations and Future Work

  • Limitations:
    • The authors note that the synthesis process might not be directly applicable to segmenting large, regular body structures that have fixed positions relative to other organs, as the generated samples might contain occluding organs not representative of real images.
  • The proposed training objective might also not be suitable for panoptic segmentation tasks due to the diversity of objects and backgrounds.
  • Future Work:
    • The authors suggest improving the image synthesis component using advanced techniques like context modeling and image harmonization.
  • Exploring its applicability to other tasks involving semi-supervised segmentation or detection of small irregular objects, such as manufacturing defect detection or remote sensing image analysis.

Summary updated

BibTeX
@inproceedings{yap2023cut,
  author = {Boon Peng Yap and Beng Koon Ng},
  title = {{Cut-Paste Consistency Learning for Semi-Supervised Lesion Segmentation}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
}
Figure