Optimizing the Dice Score and Jaccard Index for Medical Image Segmentation: Theory & Practice

Jeroen Bertels, Tom Eelbode, Maxim Berman, Dirk Vandermeulen, Frederik Maes, Raf Bisschops, Matthew B. Blaschko

TL;DR

This paper investigates the theoretical and empirical relationship between common loss functions (like cross-entropy) and evaluation metrics (Dice and Jaccard) in medical image segmentation. They demonstrate that directly optimizing metric-sensitive losses (like soft Dice/Jaccard) is superior to using cross-entropy, but the choice among different metric-sensitive losses doesn't significantly impact performance across various medical imaging tasks.

What Problem Does This Paper Address?

In medical image segmentation, the goal is to accurately delineate specific regions (e.g., tumors, organs) within images. The performance of segmentation models is typically evaluated using metrics like the Dice score and Jaccard index, which measure the overlap between the predicted segmentation and the ground truth. However, most deep learning models are trained using a loss function called cross-entropy. This creates a discrepancy: the model is optimized to minimize cross-entropy, but its performance is judged by Dice or Jaccard. This mismatch can lead to suboptimal segmentation quality. While "soft" versions of Dice and Jaccard (and other methods like Lovász-softmax) have been proposed to directly optimize these metrics, it's unclear if weighted cross-entropy can effectively approximate them, and how these different metric-sensitive losses truly compare in practice.

Methods

The authors tackle this problem from two angles: theory and empirical validation.

Theoretical Analysis: * They frame the problem within a risk minimization framework, where the goal is to minimize the expected loss. * They analyze the theoretical relationship between Dice and Jaccard indices, showing they are closely related and can approximate each other with specific bounds. * Crucially, they investigate whether a weighted cross-entropy loss can serve as an optimal surrogate for Dice or Jaccard. Their theoretical analysis suggests that no such optimal weighting exists.

Empirical Validation: * They train segmentation models using five different loss functions: * Cross-entropy (CE) * Weighted cross-entropy (wCE) * Soft Dice (sDice) * Soft Jaccard (sJaccard) * Lovász-softmax * These models are evaluated on five medical segmentation tasks. * They perform statistical significance tests to compare the performance of different loss functions.

Datasets

The study uses a combination of publicly available and in-house datasets for medical image segmentation:

  • Publicly Available 3D Datasets:
    • BRATS 2018 (limited to whole tumor segmentation)
    • ISLES 2017 (stroke lesion segmentation)
    • ISLES 2018 (acute stroke lesion segmentation)
  • In-house 2D Datasets:
    • Lower-left third molar segmentation from panoramic dental radiographs (MO17)
    • Colorectal polyp segmentation from colonoscopy images (PO18)

Key Insights and Ideas

  • Dice and Jaccard are intrinsically linked: The paper provides theoretical backing for the close relationship between Dice and Jaccard metrics, showing they can approximate each other. This means optimizing for one often leads to good performance on the other.
  • Cross-entropy is a poor surrogate for Dice/Jaccard: The core theoretical finding is that no optimal weighting of cross-entropy can effectively approximate Dice or Jaccard. This explains why models trained with cross-entropy often underperform when evaluated with these metrics.
  • Metric-sensitive losses are superior: Empirically, losses that directly optimize Dice or Jaccard (soft Dice, soft Jaccard, Lovász-softmax) consistently outperform cross-entropy-based losses. This validates the theoretical prediction.
  • Choice among metric-sensitive losses matters less than expected: Surprisingly, the study finds that the specific choice among soft Dice, soft Jaccard, and Lovász-softmax doesn't lead to statistically significant differences in performance across a range of medical segmentation tasks. This suggests that as long as you're directly optimizing the target metric, the exact flavor of the surrogate loss might be less critical.
  • Weighted cross-entropy is tricky: While weighted cross-entropy aims to address class imbalance, the paper suggests it's not a straightforward solution for approximating Dice/Jaccard and can even underperform standard cross-entropy in some cases.

Key Results

  • Empirical superiority of metric-sensitive losses: Across all five datasets, metric-sensitive losses (sDice, sJaccard, Lovász) consistently achieved higher Dice scores and Jaccard indices compared to CE and wCE.
  • No significant difference between metric-sensitive losses: Statistical tests revealed no significant difference in performance between sDice, sJaccard, and Lovász-softmax on most tasks.
  • wCE often underperforms CE: In many cases, weighted cross-entropy performed worse than standard cross-entropy, highlighting the difficulty in finding effective weights.
  • Scale-specific performance: Figure 1 shows that metric-sensitive losses generally perform better across various object sizes, especially for smaller objects, where cross-entropy struggles.

Limitations and Future Work

  • Limitations:
    • The study acknowledges that weighted cross-entropy's performance is highly task-dependent, and finding optimal weights is non-trivial.
  • While the paper shows no statistical difference between metric-sensitive losses, it's possible that for very specific tasks or datasets, one might offer a slight advantage.
  • The study focuses on binary segmentation; extending these findings to multi-class segmentation could be valuable.
  • Future Work:
    • The authors suggest that given the theoretical and empirical evidence, wider adoption of metric-sensitive losses like Dice and Jaccard is warranted in medical image segmentation research.
  • Further investigation into task-specific weighting strategies for wCE could be explored, though the paper implies this is challenging.
  • Exploring the impact of these losses on other segmentation metrics or downstream clinical tasks could be a natural extension.

Summary updated

Abstract

The Dice score and Jaccard index are commonly used metrics for the evaluation of segmentation tasks in medical imaging. Convolutional neural networks trained for image segmentation tasks are usually optimized for (weighted) cross-entropy. This introduces an adverse discrepancy between the learning optimization objective (the loss) and the end target metric. Recent works in computer vision have proposed soft surrogates to alleviate this discrepancy and directly optimize the desired metric, either through relaxations (soft-Dice, soft-Jaccard) or submodular optimization (Lov\'asz-softmax). The aim of this study is two-fold. First, we investigate the theoretical differences in a risk minimization framework and question the existence of a weighted cross-entropy loss with weights theoretically optimized to surrogate Dice or Jaccard. Second, we empirically investigate the behavior of the aforementioned loss functions w.r.t. evaluation with Dice score and Jaccard index on five medical segmentation tasks. Through the application of relative approximation bounds, we show that all surrogates are equivalent up to a multiplicative factor, and that no optimal weighting of cross-entropy exists to approximate Dice or Jaccard measures. We validate these findings empirically and show that, while it is important to opt for one of the target metric surrogates rather than a cross-entropy-based loss, the choice of the surrogate does not make a statistical difference on a wide range of medical segmentation tasks.
BibTeX
@inproceedings{bertels2019optimizing,
  author = {Jeroen Bertels and Tom Eelbode and Maxim Berman and Dirk Vandermeulen and Frederik Maes and Raf Bisschops and Matthew B. Blaschko},
  title = {{Optimizing the Dice Score and Jaccard Index for Medical Image Segmentation: Theory \& Practice}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2019},
  url = {https://openreview.net/forum?id=Qh3RY0qc92l},
}
Figure