Segmentation with Multiple Acceptable Annotations: A Case Study of Myocardial Segmentation in Contrast Echocardiography

Dewen Zeng, Mingqi Li, Yukun Ding, Xiaowei Xu, Qiu Xie, Ruixue Xu, Hongwen Fei, Meiping Huang, Jian Zhuang, Yiyu Shi

Abstract

Most existing deep learning-based frameworks for image segmentation assume that a unique ground truth is known and can be used for performance evaluation. This is true for many applications, but not all. Myocardial segmentation of Myocardial Contrast Echocardiography (MCE), a critical task in automatic myocardial perfusion analysis, is an example. Due to the low resolution and serious artifacts in MCE data, annotations from different cardiologists can vary significantly, and it is hard to tell which one is the best. In this case, how can we find a good way to evaluate segmentation performance and how do we train the neural network? In this paper, we address the first problem by proposing a new extended Dice to effectively evaluate the segmentation performance when multiple accepted ground truth is available. Then based on our proposed metric, we solve the second problem by further incorporating the new metric into a loss function that enables neural networks to flexibly learn general features of myocardium. Experiment results on our clinical MCE data set demonstrate that the neural network trained with the proposed loss function outperforms those existing ones that try to obtain a unique ground truth from multiple annotations, both quantitatively and qualitatively. Finally, our grading study shows that using extended Dice as an evaluation metric can better identify segmentation results that need manual correction compared with using Dice.

Introduction

In medical imaging, the concept of a single "ground truth" is often a simplification of a more complex reality. For tasks like Myocardial Contrast Echocardiography (MCE) segmentation, images frequently suffer from low resolution, noise, and artifacts. Consequently, even experienced cardiologists may disagree on the precise boundaries of the myocardium. Traditional deep learning approaches force models to converge toward a single, often arbitrary, gold standard—such as a majority vote or a single expert's annotation. This paper argues that such constraints introduce noise into the training process, hindering the model's ability to learn robust anatomical features.

Methodology

The Extended Dice Metric

To accommodate inter-observer variability, the authors propose the Extended Dice Metric. Instead of a single boundary, they define an "Acceptable Region" bounded by two limits:

  • Inner Boundary ($I$): The intersection of all expert annotations, representing the area where all experts agree the myocardium exists.
  • Outer Boundary ($O$): The union of all expert annotations, representing the maximum extent any expert considered to be myocardium.
Comparison of multiple expert annotations (a, b) and resulting segmentation regions (c, d) in MCE images.

The Extended Dice metric only penalizes predictions that fall outside this acceptable region—either by missing the inner boundary or exceeding the outer boundary. This provides a more realistic evaluation framework for ambiguous medical data.

Extended Dice Loss

The metric is further converted into a differentiable loss function, allowing it to be used directly during the training of neural networks (e.g., U-Net or DeepLab). This "flexibility" allows the network to focus on learning general textures and anatomical consensus rather than being penalized for valid variations in expert judgment. In clear regions of an image, the acceptable region is narrow, requiring high precision; in noisy regions, the region widens, allowing the model to prioritize general feature learning.

Experimental Setup and Datasets

The proposed methods were evaluated on a specialized clinical MCE dataset consisting of 1,000 images from 100 subjects (40 with coronary artery disease and 60 healthy controls). Crucially, each image was manually labeled by five experienced cardiologists, providing the multi-annotator data necessary for the Extended Dice framework. This dataset has been made available by the authors as a benchmark for the research community.

Key Results and Discussion

Robustness and Evaluation Accuracy

The model trained with Extended Dice loss consistently outperformed baseline models trained on single-expert annotations or majority votes. Furthermore, the Extended Dice metric proved superior at identifying AI-generated segmentations that required manual correction, achieving an AUC of 0.94 compared to 0.86 for the standard Dice metric.

Clinical Utility

A blind grading study by an independent cardiologist found that the Extended Dice approach produced the highest number of "Excellent" segmentations and the fewest "Severe Mistakes" compared to state-of-the-art methods like STAPLE. Additionally, the resulting frame-intensity curves—a vital tool for diagnosing heart disease—more closely matched the expert average than models trained on individual ground truths.

Conclusion

By embracing the inherent ambiguity of expert annotations rather than forcing a false consensus, the Extended Dice framework allows neural networks to learn more robust and clinically relevant features. While currently focused on MCE, this approach holds significant potential for other medical imaging modalities where a single, definitive ground truth is unattainable.

Summary updated

Abstract

Most existing deep learning-based frameworks for image segmentation assume that a unique ground truth is known and can be used for performance evaluation. This is true for many applications, but not all. Myocardial segmentation of Myocardial Contrast Echocardiography (MCE), a critical task in automatic myocardial perfusion analysis, is an example. Due to the low resolution and serious artifacts in MCE data, annotations from different cardiologists can vary significantly, and it is hard to tell which one is the best. In this case, how can we find a good way to evaluate segmentation performance and how do we train the neural network? In this paper, we address the first problem by proposing a new extended Dice to effectively evaluate the segmentation performance when multiple accepted ground truth is available. Then based on our proposed metric, we solve the second problem by further incorporating the new metric into a loss function that enables neural networks to flexibly learn general features of myocardium. Experiment results on our clinical MCE data set demonstrate that the neural network trained with the proposed loss function outperforms those existing ones that try to obtain a unique ground truth from multiple annotations, both quantitatively and qualitatively. Finally, our grading study shows that using extended Dice as an evaluation metric can better identify segmentation results that need manual correction compared with using Dice.
BibTeX
@inproceedings{zeng2021segmentation,
  author = {Dewen Zeng and Mingqi Li and Yukun Ding and Xiaowei Xu and Qiu Xie and Ruixue Xu and Hongwen Fei and Meiping Huang and Jian Zhuang and Yiyu Shi},
  title = {{Segmentation with Multiple Acceptable Annotations: A Case Study of Myocardial Segmentation in Contrast Echocardiography}},
  booktitle = {IPMI},
  year = {2021},
  url = {https://openreview.net/forum?id=EvA2ymyfmx},
}
Figure