Learning to Segment Skin Lesions from Noisy Annotations

Zahra Mirikharaji, Yiqi Yan, Ghassan Hamarneh

Abstract

Training deep neural networks for medical image segmentation typically requires a massive amount of pixel-perfect expert annotations, which are expensive and time-consuming to produce. This paper introduces a meta-learning framework designed to learn effectively from a large pool of "noisy" (imperfect) labels by utilizing a tiny set of "clean" expert labels. The framework dynamically reweights the importance of every pixel during training through a meta-learning loop. Experimental results on the ISIC 2017 dataset demonstrate that this approach significantly improves performance with minimal clean data, even outperforming standard full supervision in certain configurations.

Introduction

Deep learning has revolutionized medical imaging, yet the requirement for high-quality, pixel-level annotations remains a significant bottleneck. While non-experts or automated tools can generate "noisy" annotations quickly, deep networks tend to memorize this noise, leading to poor generalization. This paper addresses the challenge of leveraging a small amount of expensive, high-quality data alongside a large volume of cheap, low-quality data to build robust segmentation models. Unlike previous image-level reweighting methods, this work proposes a spatially adaptive, pixel-level reweighting strategy.

Methodology: Spatially Adaptive Reweighting

The core of the proposed method is a meta-learning loop that treats the training process as a nested optimization problem. Instead of assuming every pixel in a noisy mask is correct, the model learns a weight map for each image.

The process follows four main steps:

  1. The Noisy Step: A standard U-Net model takes a tentative gradient step based on a batch of noisy data.
  2. The Meta-Evaluation: The model evaluates its performance on a small batch of "clean" expert data. It calculates how the loss on this clean data would change if the weights of the pixels in the noisy batch were adjusted.
  3. The Reweighting: If a noisy pixel's gradient aligns with the clean data's gradient, it receives a higher weight. Conversely, if a pixel leads the model astray, its weight is pushed toward zero.
  4. The Final Update: The model performs its actual training update using these optimized, spatially-aware weights.

This end-to-end process allows the model to simultaneously learn the segmentation task and identify which parts of the noisy labels are trustworthy.

Experimental Setup

The researchers evaluated their method using the ISIC 2017 (International Skin Imaging Collaboration) dataset, consisting of 2,000 training images of skin lesions. To simulate real-world noise, expert masks were algorithmically simplified into various geometric forms.

Comparison of original image, ground truth (GT), and various simulated noisy labels used in the study.

The noise types included:

  • 7-vertex, 3-vertex, and 4-axis-aligned polygons: Simplified versions of the expert-vetted masks.
  • Unsupervised: A generic rectangle covering most of the image, representing extreme noise.

Results and Analysis

The proposed framework demonstrated remarkable efficiency and robustness across several metrics:

  • High Performance with Minimal Data: Using just 10 clean images to guide learning from 1,990 noisy ones resulted in a Dice score improvement of 21.79% compared to training on noisy data alone.
  • Outperforming Full Supervision: With 100 clean images, the method achieved a Dice score of approximately 80%, surpassing a standard U-Net trained on the full set of 2,000 clean images. This suggests that the adaptive weighting acts as a powerful regularizer, helping the model generalize better than standard training.
  • Robustness to Extreme Noise: The model maintained a high Dice score (~79%) even when noisy labels were extremely crude (e.g., 3-vertex polygons), whereas standard fine-tuning approaches suffered significant performance drops.

Discussion and Conclusion

The success of this method highlights the importance of pixel-level granularity in segmentation tasks. By focusing on gradients that align with clean data, the model can ignore messy boundaries in noisy masks while still learning from the correct interior regions.

However, there are trade-offs. The meta-learning loop introduces computational overhead, requiring multiple forward and backward passes per iteration. Additionally, while the requirement for clean data is minimal, it is not zero. Future work could explore combining this framework with different architectures or extending it to handle inter-observer variability between multiple experts.

In conclusion, spatially adaptive reweighting offers a viable path for training high-performance medical segmentation models in data-constrained environments, effectively bridging the gap between cheap noisy labels and expensive expert annotations.

Summary updated

Abstract

Deep convolutional neural networks have driven substantial advancements in the automatic understanding of images. Requiring a large collection of images and their associated annotations is one of the main bottlenecks limiting the adoption of deep networks. In the task of medical image segmentation, requiring pixel-level semantic annotations performed by human experts exacerbate this difficulty. This paper proposes a new framework to train a fully convolutional segmentation network from a large set of cheap unreliable annotations and a small set of expert-level clean annotations. We propose a spatially adaptive reweighting approach to treat clean and noisy pixel-level annotations commensurately in the loss function. We deploy a meta-learning approach to assign higher importance to pixels whose loss gradient direction is closer to those of clean data. Our experiments on training the network using segmentation ground truth corrupted with different levels of annotation noise show how spatial reweighting improves the robustness of deep networks to noisy annotations.
BibTeX
@inproceedings{mirikharaji2019learning,
  author = {Zahra Mirikharaji and Yiqi Yan and Ghassan Hamarneh},
  title = {{Learning to Segment Skin Lesions from Noisy Annotations}},
  booktitle = {DART/MIL3ID @ MICCAI},
  year = {2019},
  doi = {10.1007/978-3-030-33391-1\_24},
}
Figure