Optimizing the Dice Score and Jaccard Index for Medical Image Segmentation: Theory and Practice

Jeroen Bertels, Tom Eelbode, Maxim Berman, Dirk Vandermeulen, Frederik Maes, Raf Bisschops, Matthew B. Blaschko

TL;DR

This paper investigates the theoretical and practical implications of using different loss functions for medical image segmentation, specifically comparing standard cross-entropy with metric-sensitive losses like soft Dice and soft Jaccard. They demonstrate that while metric-sensitive losses generally outperform cross-entropy, the choice among different metric-sensitive losses often doesn't lead to statistically significant differences in performance.

What Problem Does This Paper Address?

In medical image segmentation, the goal is to accurately delineate specific structures (e.g., tumors, organs). While the Dice score and Jaccard index are the standard metrics for evaluating segmentation quality, most deep learning models are trained using cross-entropy loss. This creates a mismatch: the model is optimized to minimize one objective (cross-entropy) while its performance is judged by another (Dice/Jaccard). This discrepancy can lead to suboptimal segmentation results. Recent research has proposed "soft" or differentiable versions of Dice and Jaccard (e.g., soft Dice, soft Jaccard, Lovász-softmax) to directly optimize these metrics during training. However, the theoretical underpinnings of this optimization and the practical benefits of choosing one metric-sensitive loss over another, especially compared to weighted cross-entropy, remain unclear.

Methods

The authors tackle this problem from two angles: theoretical analysis and empirical validation.

Theoretical Analysis: * They frame the problem within a risk minimization framework, where the goal is to minimize the expected loss. * They analyze the theoretical relationship between the Dice score, Jaccard index, and weighted Hamming similarity (a proxy for weighted cross-entropy). * They derive approximation bounds to understand how well different loss functions can approximate the target metrics.

Empirical Validation: * They train convolutional neural networks (CNNs) using five different loss functions: cross-entropy (CE), weighted cross-entropy (wCE), soft Dice (sDice), soft Jaccard (sJaccard), and Lovász-softmax. * They evaluate these models on five medical segmentation tasks using both Dice and Jaccard scores. * Statistical significance tests (bootstrapping) are used to compare the performance of different loss functions. * They also conduct a scale-specific study to investigate how loss functions perform on objects of varying sizes.

Datasets

The study uses a combination of publicly available and in-house datasets:

  • Publicly Available 3D Datasets:
    • BRATS 2018 (whole tumor segmentation)
    • ISLES 2017 (stroke lesion segmentation)
    • ISLES 2018 (acute stroke lesion segmentation)
  • In-house 2D Datasets:
    • MO17 (lower-left third molar segmentation from panoramic dental radiographs)
    • PO18 (colorectal polyp segmentation from colonoscopy images)

Key Insights and Ideas

  • Dice and Jaccard are intrinsically linked: Theoretically, the Dice score and Jaccard index are closely related and can approximate each other. This means optimizing for one often leads to good performance on the other.
  • Weighted cross-entropy is not a perfect surrogate: The paper theoretically shows that no optimal weighting of cross-entropy can perfectly approximate the Dice or Jaccard measures. This is a crucial finding, as weighted cross-entropy is often used to handle class imbalance.
  • Metric-sensitive losses are generally superior: Empirically, losses that directly optimize for Dice or Jaccard (sDice, sJaccard, Lovász-softmax) consistently outperform standard cross-entropy and weighted cross-entropy. This confirms the theoretical mismatch.
  • Choice among metric-sensitive losses is less critical: While metric-sensitive losses are better than cross-entropy, the study found that the choice between sDice, sJaccard, and Lovász-softmax often does not lead to statistically significant differences in performance across various medical segmentation tasks. This simplifies the decision-making process for practitioners.
  • Scale matters: The performance of different loss functions can vary depending on the size of the objects being segmented. While metric-sensitive losses generally perform well across scales, cross-entropy can struggle, especially with smaller objects.

Key Results

  • Performance Improvement: Metric-sensitive losses (sDice, sJaccard, Lovász) consistently achieved higher Dice scores and Jaccard indexes compared to CE and wCE across most datasets.
  • No Significant Difference Among Metric-Sensitive Losses: Statistical tests revealed no significant difference in performance between sDice, sJaccard, and Lovász-softmax on average across the tested datasets.
  • Weighted Cross-Entropy Underperformance: In many cases, wCE performed worse than standard CE, suggesting that simple re-weighting is not a robust solution for approximating Dice/Jaccard.
  • Scale-Specific Behavior: Figure 1 shows that metric-sensitive losses generally maintain higher Dice scores across a range of object sizes compared to CE, especially for smaller objects.

Limitations and Future Work

  • Weighted Cross-Entropy Weighting: The paper acknowledges that finding optimal weights for wCE is task-dependent and non-trivial. While their heuristic approach didn't yield superior results, better weighting strategies might exist.
  • Scale-Specific Optimization: While the study highlights scale-dependent performance, it doesn't propose specific methods for optimizing losses that are inherently scale-aware beyond the general metric-sensitive losses.
  • Limited Number of Tasks: While five diverse tasks were used, exploring a wider range of medical imaging modalities and segmentation challenges could further validate the findings.
  • Future Work: The authors suggest that the theoretical insights and empirical results warrant wider adoption of metric-sensitive losses in medical image segmentation, moving away from the prevalent use of per-pixel cross-entropy. Further research could focus on developing more sophisticated scale-aware loss functions or exploring the interplay between different loss functions and network architectures.

Summary updated

Abstract

The Dice score and Jaccard index are commonly used metrics for the evaluation of segmentation tasks in medical imaging. Convolutional neural networks trained for image segmentation tasks are usually optimized for (weighted) cross-entropy. This introduces an adverse discrepancy between the learning optimization objective (the loss) and the end target metric. Recent works in computer vision have proposed soft surrogates to alleviate this discrepancy and directly optimize the desired metric, either through relaxations (soft-Dice, soft-Jaccard) or submodular optimization (Lovász-softmax). The aim of this study is two-fold. First, we investigate the theoretical differences in a risk minimization framework and question the existence of a weighted cross-entropy loss with weights theoretically optimized to surrogate Dice or Jaccard. Second, we empirically investigate the behavior of the aforementioned loss functions w.r.t. evaluation with Dice score and Jaccard index on five medical segmentation tasks. Through the application of relative approximation bounds, we show that all surrogates are equivalent up to a multiplicative factor, and that no optimal weighting of cross-entropy exists to approximate Dice or Jaccard measures. We validate these findings empirically and show that, while it is important to opt for one of the target metric surrogates rather than a cross-entropy-based loss, the choice of the surrogate does not make a statistical difference on a wide range of medical segmentation tasks.
BibTeX
@inproceedings{bertels2019optimizing,
  author = {Jeroen Bertels and Tom Eelbode and Maxim Berman and Dirk Vandermeulen and Frederik Maes and Raf Bisschops and Matthew B. Blaschko},
  title = {{Optimizing the Dice Score and Jaccard Index for Medical Image Segmentation: Theory and Practice}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2019},
  url = {https://openreview.net/forum?id=Qh3RY0qc92l},
}
Figure