Segmentation with Multiple Acceptable Annotations: A Case Study of Myocardial Segmentation in Contrast Echocardiography

Dewen Zeng, Mingqi Li, Yukun Ding, Xiaowei Xu, Qiu Xie, Ruixue Xu, Hongwen Fei, Meiping Huang, Jian Zhuang, Yiyu Shi

TL;DR

In medical imaging, experts often disagree on where a boundary lies, yet standard AI training forces models to pick a single "correct" version. This paper introduces Extended Dice, a new metric and loss function that defines an "acceptable region" between multiple expert annotations, allowing neural networks to learn general features more effectively without being penalized for valid inter-observer variations.

What Problem Does This Paper Address?

Most deep learning frameworks for image segmentation rely on a "ground truth"—a single, definitive gold standard. However, in complex medical tasks like Myocardial Contrast Echocardiography (MCE), the "truth" is blurry. MCE images suffer from low resolution, noise, and artifacts. When five different cardiologists annotate the same heart wall, they produce five different shapes.

Traditional methods try to solve this by taking a majority vote or picking one "best" expert. The authors argue this is flawed: if the image quality is poor, even experts can't agree, and a majority vote might result in an irregular, non-anatomical boundary. Forcing a model to mimic one specific expert's arbitrary choice in a noisy region introduces "noise" into the training process, hindering the model's ability to learn the actual underlying anatomy.

Methods

The authors propose two major innovations:

  1. Extended Dice Metric: Instead of comparing a prediction to one line, they define two boundaries based on multiple experts:
  2. Inner Boundary ($I$): The intersection of all expert annotations (where everyone agrees it's myocardium).
  3. Outer Boundary ($O$): The union of all expert annotations (the maximum extent anyone considered myocardium).
  4. The area between $I$ and $O$ is the Acceptable Region. The Extended Dice metric only penalizes the model if it misses the inner boundary or exceeds the outer boundary. Anything inside the acceptable region is considered "correct."
  5. Extended Dice Loss: They converted this metric into a differentiable loss function for training. This gives the neural network "flexibility" during training. If the image is clear, the acceptable region is narrow, and the model must be precise. If the image is noisy and experts disagree, the region is wide, and the model can focus on learning general textures rather than obsessing over a specific pixel-perfect line that might be wrong.

The researchers tested this approach using two popular architectures: U-Net and DeepLab.

Datasets

The study uses a specialized clinical dataset:

  • MCE Dataset: 1,000 images from 100 subjects (40 with coronary artery disease, 60 healthy).
  • Annotations: Each image was manually labeled by five experienced cardiologists.
  • Availability: The authors have made this dataset available online, providing a rare benchmark for the MCE community.

Key Insights and Ideas

  • Flexibility is a Feature, Not a Bug: By not forcing the model to choose between experts, the model actually learns better features. It ignores the "human factor" noise of individual annotators and focuses on the consensus of the anatomy.
  • Metric-Loss Alignment: Training a model on the same metric used to evaluate it (Extended Dice) leads to more robust performance than using standard Cross-Entropy.
  • Clinical Relevance: The authors didn't just look at pixel accuracy; they looked at the frame-intensity curve (a key clinical tool for diagnosing heart disease). Their method produced curves that more closely matched the expert average than any single-expert model.

Key Results

  • Superior Robustness: When evaluated against any individual cardiologist's "truth," the model trained with Extended Dice loss consistently outperformed models trained on single experts or majority votes.
  • Better Evaluation: The Extended Dice metric was better at identifying which AI segmentations actually needed manual correction by a human. It achieved an AUC of 0.94 for this classification task, compared to 0.86 for the standard Dice metric.
  • Qualitative Quality: In a blind "grading study," an independent cardiologist rated the Extended Dice results as having the highest number of "Excellent" (Level 4) segmentations and the fewest "Severe Mistake" (Level 1) cases compared to state-of-the-art methods like STAPLE or Confusion Matrix modeling.

Limitations and Future Work

  • Scope: The paper focuses specifically on MCE. While the logic of "acceptable regions" applies to many medical fields (like CT or MRI), the paper does not demonstrate results on other modalities.
  • Expert Requirement: The method requires multiple annotations per image for training. In many clinical settings, getting five experts to label the same image is a luxury that may not be scalable without pre-existing multi-annotator datasets.
  • Future Directions: The authors suggest this approach could be applied to other "ambiguous" segmentation problems where a single ground truth is fundamentally unattainable.

Summary updated

Abstract

Most existing deep learning-based frameworks for image segmentation assume that a unique ground truth is known and can be used for performance evaluation. This is true for many applications, but not all. Myocardial segmentation of Myocardial Contrast Echocardiography (MCE), a critical task in automatic myocardial perfusion analysis, is an example. Due to the low resolution and serious artifacts in MCE data, annotations from different cardiologists can vary significantly, and it is hard to tell which one is the best. In this case, how can we find a good way to evaluate segmentation performance and how do we train the neural network? In this paper, we address the first problem by proposing a new extended Dice to effectively evaluate the segmentation performance when multiple accepted ground truth is available. Then based on our proposed metric, we solve the second problem by further incorporating the new metric into a loss function that enables neural networks to flexibly learn general features of myocardium. Experiment results on our clinical MCE data set demonstrate that the neural network trained with the proposed loss function outperforms those existing ones that try to obtain a unique ground truth from multiple annotations, both quantitatively and qualitatively. Finally, our grading study shows that using extended Dice as an evaluation metric can better identify segmentation results that need manual correction compared with using Dice.
BibTeX
@inproceedings{zeng2021segmentation,
  author = {Dewen Zeng and Mingqi Li and Yukun Ding and Xiaowei Xu and Qiu Xie and Ruixue Xu and Hongwen Fei and Meiping Huang and Jian Zhuang and Yiyu Shi},
  title = {{Segmentation with Multiple Acceptable Annotations: A Case Study of Myocardial Segmentation in Contrast Echocardiography}},
  booktitle = {IPMI},
  year = {2021},
  url = {https://openreview.net/forum?id=EvA2ymyfmx},
}

Loading related content...

Figure