Optimization for Medical Image Segmentation: Theory and Practice When Evaluating With Dice Score or Jaccard Index

Tom Eelbode, Jeroen Bertels, Maxim Berman, Dirk Vandermeulen, Frederik Maes, Raf Bisschops, Matthew B. Blaschko

TL;DR

This paper investigates why many medical image segmentation models are trained with loss functions that don't directly optimize the commonly used Dice or Jaccard metrics. Through theoretical analysis and extensive experiments, they demonstrate that metric-sensitive losses (like soft Dice) consistently outperform traditional cross-entropy losses for these evaluation metrics, encouraging their wider adoption.

What Problem Does This Paper Address?

In medical image segmentation, the Dice score and Jaccard index are the gold standard for evaluating how well a model's prediction matches the ground truth. However, many researchers still train their models using simpler, per-pixel loss functions like cross-entropy. This creates a disconnect: the model is optimized for one thing (pixel-wise accuracy) but evaluated on another (overlap metrics). The authors aim to understand the theoretical implications of this mismatch and empirically validate whether using loss functions that directly approximate the Dice or Jaccard index leads to better performance.

Methods

The paper combines theoretical analysis with empirical validation.

  • Theoretical Analysis: They investigate the mathematical relationships between various similarity metrics (Dice, Jaccard, Tversky, Hamming) and their differentiable "soft" approximations used in training. They analyze how well these soft losses can approximate the true discrete metrics, particularly focusing on whether cross-entropy can serve as a surrogate.
  • Empirical Validation: They train Convolutional Neural Networks (CNNs) on six diverse medical image segmentation datasets using several loss functions:
  • Cross-Entropy (CE)
    • Weighted Cross-Entropy (wCE)
    • Soft Dice (sDice)
    • Soft Jaccard (sJaccard)
    • Lovász-softmax (Lovász)
    • Soft Tversky (sTversky) They then compare the performance of these models using Dice and Jaccard scores, as well as other metrics like accuracy and Hausdorff distance. They also explore the impact of class imbalance and object size.

Datasets

The study utilizes six publicly available and in-house medical image segmentation datasets:

  • BRATS 2018: Brain tumor segmentation (3D MRI).
  • ISLES 2017: Ischemic stroke lesion segmentation (3D acute MRI perfusion).
  • ISLES 2018: Ischemic core segmentation (3D acute CT perfusion).
  • MO17: Lower-left third molar segmentation (2D panoramic dental radiographs).
  • PO18: Colorectal polyp segmentation (2D colonoscopy images).
  • WMH 2017: White matter hyperintensities segmentation (3D MRI).

The paper also describes creating virtual datasets from PO18 to specifically investigate the influence of foreground-background ratios.

Key Insights and Ideas

  • The "Metric Mismatch" is Real and Detrimental: The core theoretical finding is that cross-entropy (and its weighted variant) does not approximate the Dice or Jaccard index. This means optimizing for cross-entropy doesn't guarantee good performance on these crucial overlap metrics, and can even lead to arbitrarily poor results.
  • Metric-Sensitive Losses are Superior: Soft versions of Dice, Jaccard, and Lovász-softmax are shown to be effective surrogates for their respective metrics. They consistently outperform cross-entropy-based losses when the evaluation metric is Dice or Jaccard.
  • Dice and Jaccard are Closely Related: The paper confirms the theoretical equivalence and strong empirical correlation between Dice and Jaccard scores. Optimizing for one generally leads to similar performance for the other.
  • Tversky's Flexibility Comes with a Trade-off: While the Tversky loss offers tunable parameters (alpha and beta), its approximation to the Dice score degrades as these parameters deviate from 0.5 (which makes it equivalent to soft Dice). For optimizing Dice score, the 0.5/0.5 weighting is indeed optimal.
  • Class Imbalance and Object Size Matter, but Metric-Sensitive Losses Still Win: The study shows that metric-sensitive losses are beneficial across a wide range of object sizes and foreground-background ratios, often outperforming CE even in challenging scenarios like small objects or low fg/bg ratios.

Key Results

  • Consistent Performance Gains: Across all six datasets and various configurations (including multi-class segmentation, different object sizes, and fg/bg ratios), metric-sensitive losses (sDice, sJaccard, Lovász) consistently achieved higher Dice and Jaccard scores compared to CE and wCE.
  • No Significant Difference Among Metric-Sensitive Losses: For Dice/Jaccard evaluation, sDice, sJaccard, and Lovász performed similarly well, suggesting a free choice among these for practical applications.
  • Tversky Optimal at α=β=0.5: For optimizing Dice score, the Tversky loss performs best when α=β=0.5, making it equivalent to soft Dice. Deviating from this weighting leads to degraded performance.
  • Qualitative Improvements: Visual inspection of segmentation results (Figure 3) showed that metric-sensitive losses produced more consistent and accurate segmentations with narrower output value ranges, whereas CE often under-segmented structures.
  • No Significant Difference for Other Metrics: While metric-sensitive losses excel for Dice/Jaccard, their performance on metrics like Accuracy (ACC) was sometimes inferior to CE, highlighting that the choice of loss function is tied to the evaluation metric. However, for Hausdorff Distance (HDD) and Absolute Volume Difference (AVD), sDice was often comparable or better than CE.

Limitations and Future Work

  • Focus on Dice/Jaccard: The study primarily focuses on optimizing for Dice and Jaccard. While they briefly touch upon other metrics like ACC, HDD, and AVD, the main conclusion is geared towards scenarios where Dice/Jaccard are the primary evaluation metrics.
  • Generalizability to Other Metrics: The paper acknowledges that other metrics (like distance-based ones) might be of interest, and while they show some results, a comprehensive analysis across all possible evaluation metrics is beyond the scope.
  • Weighting of Cross-Entropy: The paper demonstrates that finding an optimal weighting for wCE is non-trivial and application-dependent, suggesting it's unlikely to find a universal weighting that makes it a good surrogate for Dice/Jaccard.
  • Future Work: The authors encourage wider adoption of metric-sensitive loss functions for tasks where Dice or Jaccard are the primary performance measures. They also suggest that further research could explore combinations of loss functions to optimize for multiple metrics simultaneously.

Summary updated

Abstract

In many medical imaging and classical computer vision tasks, the Dice score and Jaccard index are used to evaluate the segmentation performance. Despite the existence and great empirical success of metric-sensitive losses, i.e. relaxations of these metrics such as soft Dice, soft Jaccard and Lovász-Softmax, many researchers still use per-pixel losses, such as (weighted) cross-entropy to train CNNs for segmentation. Therefore, the target metric is in many cases not directly optimized. We investigate from a theoretical perspective, the relation within the group of metric-sensitive loss functions and question the existence of an optimal weighting scheme for weighted cross-entropy to optimize the Dice score and Jaccard index at test time. We find that the Dice score and Jaccard index approximate each other relatively and absolutely, but we find no such approximation for a weighted Hamming similarity. For the Tversky loss, the approximation gets monotonically worse when deviating from the trivial weight setting where soft Tversky equals soft Dice. We verify these results empirically in an extensive validation on six medical segmentation tasks and can confirm that metric-sensitive losses are superior to cross-entropy based loss functions in case of evaluation with Dice Score or Jaccard Index. This further holds in a multi-class setting, and across different object sizes and foreground/background ratios. These results encourage a wider adoption of metric-sensitive loss functions for medical segmentation tasks where the performance measure of interest is the Dice score or Jaccard index.
BibTeX
@article{eelbode2020optimization,
  author = {Tom Eelbode and Jeroen Bertels and Maxim Berman and Dirk Vandermeulen and Frederik Maes and Raf Bisschops and Matthew B. Blaschko},
  title = {{Optimization for Medical Image Segmentation: Theory and Practice When Evaluating With Dice Score or Jaccard Index}},
  journal = {IEEE Trans. Medical Imaging},
  year = {2020},
  url = {https://openreview.net/forum?id=YnxF5tB9wvm},
}
Figure