Active learning of the ground truth for retinal image segmentation

Liubov Nedoshivina, Lasse LensuJournal of Optical Technology 2019 · DOI

TL;DR

This paper tackles the challenge of efficiently creating accurate ground truth data for retinal image segmentation by employing active learning. By intelligently selecting which images to annotate, the authors demonstrate a way to achieve high performance with significantly less manual labeling effort compared to traditional methods.

What Problem Does This Paper Address?

Diagnosing eye diseases from retinal fundus images is crucial, and automated methods can greatly assist medical professionals. However, developing these automated systems requires large, accurately annotated datasets. Creating this "ground truth" data is a time-consuming and expensive process, as it typically involves multiple medical experts meticulously labeling features like blood vessels in thousands of images. This paper addresses the inefficiency of this manual annotation process by proposing and evaluating active learning strategies to accelerate the creation of high-quality ground truth data for retinal image segmentation.

Methods

The core of this work lies in applying active learning to streamline the ground truth annotation process for retinal image segmentation. Two primary segmentation methods were investigated:

  • Soares Method: This approach utilizes Gabor features and a supervised Bayesian classifier. The key innovation here is using the log-likelihood (LL) of segmented images as an uncertainty measure. Active learning then selects images with the highest LL for expert annotation, aiming to improve the model's understanding of uncertain regions.
  • U-Net CNN Architecture: A popular deep learning model for image segmentation, U-Net was also adapted for active learning. Here, the authors explored using neuron activations and Monte Carlo Dropout during testing to estimate pixel-wise uncertainty. The most uncertain samples were then selected for annotation.

For both methods, the active learning process involves iteratively training the model, using its current performance to identify the most informative unlabeled images, having those images annotated by experts, and then retraining the model with the newly labeled data. This contrasts with standard approaches where all data is labeled upfront.

Datasets

The study utilized several publicly available retinal image datasets for training and evaluation:

  • DRIVE: A commonly used dataset for retinal vessel segmentation.
  • STARE: Another established dataset for retinal image analysis.
  • CHASEDB1: A dataset containing color images of the retina.
  • ARIADB: A dataset with a relatively large number of manually segmented retinal images.

The paper does not introduce new benchmark datasets but rather focuses on applying active learning to existing ones.

Key Insights and Ideas

  • Active Learning Significantly Reduces Annotation Effort: The most striking insight is that active learning can achieve performance comparable to fully trained models with substantially fewer annotated images. This has direct implications for reducing the cost and time associated with developing medical imaging AI.
  • Uncertainty is Key: The effectiveness of active learning hinges on accurately identifying which images the model is most uncertain about. The paper demonstrates that different uncertainty measures (log-likelihood for Soares, dropout-based variance for U-Net) can successfully guide the selection of informative samples.
  • Early Convergence is Possible: The Soares model, in particular, showed rapid improvement, reaching near-optimal performance within 4-6 active learning iterations. This suggests that active learning can quickly bootstrap model performance.
  • Pretraining Enhances U-Net with Limited Data: For the U-Net, pretraining on smaller image patches proved beneficial, allowing the model to learn effectively even with a limited initial labeled dataset.

Key Results

  • Soares Model Performance: The Soares model, when trained with active learning, achieved performance levels close to the fully trained model in 4-6 iterations. The best performance and learning rate were observed on the DRIVE dataset. For instance, on the DRIVE dataset, active learning with uncertainty sampling resulted in segmentation quality (DC) that closely matched the fully trained model after a few iterations.
  • U-Net Model Performance: While specific quantitative results for U-Net are less detailed in the provided text, the paper indicates that active learning allowed the U-Net to achieve performance levels comparable to models trained on larger datasets, even when starting with a small labeled set. Visual results showed that active learning improved segmentation quality, particularly in capturing finer vessel structures.

Limitations and Future Work

  • Query Function Accuracy: The authors suggest that future research could improve the query function's accuracy by incorporating a mask image that focuses the uncertainty calculation specifically on the retinal region of interest.
  • Dataset Homogenization: Combining multiple retinal image databases into a single, larger dataset could provide more information for model training. However, this requires careful preprocessing to ensure homogeneity in terms of color, illumination, and resolution across different datasets.
  • Generalizability: While the study explored active learning on several datasets, further investigation into its generalizability across a wider range of retinal image qualities and disease presentations would be valuable.

Summary updated

Abstract

Different diseases can be diagnosed from eye fundus images by medical experts. Automated diagnosis methods can help medical doctors to increase the diagnosis accuracy and decrease the time needed. In order to have a proper dataset for training and evaluating the methods, a large set of images should be annotated by several experts to form the ground truth. To enable efficient utilization of the experts’ time, active learning is studied to accelerate the collection of the ground truth. Since one of the important steps in retinal image diagnosis is blood vessel segmentation, the corresponding approaches were studied. Two approaches were implemented and extended by proposed active learning methods for selecting the next image to be annotated. The performance of the methods in the cases of standard implementation and active learning application was compared for several retinal image databases.
BibTeX
@article{nedoshivina2019active,
  author = {Liubov Nedoshivina and Lasse Lensu},
  title = {{Active learning of the ground truth for retinal image segmentation}},
  journal = {Journal of Optical Technology},
  year = {2019},
  doi = {10.1364/jot.86.000697},
}
Figure