Optimization of the Jaccard index for image segmentation with the Lovász hinge

Maxim Berman, Matthew B. BlaschkoarXiv 2017 · arXiv · OpenReview

TL;DR

This paper introduces the "Lovász-Softmax loss," a novel method for directly optimizing the Jaccard index (Intersection-over-Union or IoU) in neural networks for semantic image segmentation. By leveraging the convex Lovász extension of submodular functions, this loss function leads to significantly improved segmentation performance compared to traditional cross-entropy loss, particularly for small objects and overall segmentation quality.

What Problem Does This Paper Address?

Semantic image segmentation aims to classify every pixel in an image into a specific object category. While cross-entropy loss is a common choice for training segmentation models, it doesn't directly optimize the most widely used evaluation metric: the Jaccard index (IoU). IoU is preferred because it's more perceptually relevant, handles scale invariance well, and is better at counting false negatives than pixel-wise losses. However, directly optimizing IoU is challenging because it's a non-differentiable, discrete metric. Existing methods for optimizing IoU have limitations, often requiring complex approximations, structured output models, or not being easily pluggable into modern deep learning architectures. This paper addresses the need for a direct, differentiable, and easily integrated loss function that optimizes IoU for semantic segmentation.

Methods

The core contribution is the development of the Lovász-Softmax loss. This method builds upon the concept of the Lovász extension, a technique for creating convex, differentiable surrogates for submodular set functions. The Jaccard index loss is shown to be submodular, making it amenable to this approach.

Here's a breakdown of the key methodological aspects:

  • Lovász Extension: The authors leverage the Lovász extension to transform the discrete Jaccard loss into a continuous, convex function. This allows for gradient-based optimization in neural networks.
  • Lovász Hinge (Binary Case): For binary segmentation (foreground/background), they adapt the Lovász hinge loss, which is a direct convex surrogate for the Jaccard loss.
  • Lovász-Softmax (Multiclass Case): For multiclass segmentation, they extend this by incorporating a softmax activation layer. This maps the network's raw scores to probabilities, and then applies the Lovász extension to these probabilities to create the Lovász-Softmax loss. This loss is defined as the average of class-specific Jaccard losses, weighted by the network's predicted probabilities.
  • "Equibatch" Strategy: To better align with the dataset-level IoU metric, they propose an "equibatch" training strategy. This involves sampling training data such that each class is represented in the minibatch over a cycle, making the loss more robust to classes absent in individual minibatches.
  • Proximal Gradient Algorithm: For optimization, they discuss the use of proximal gradient algorithms, which are well-suited for piecewise linear functions like the Lovász extension, offering potentially faster convergence than standard gradient descent.

Datasets

The paper evaluates their method on two prominent semantic segmentation datasets:

  • Pascal VOC: A widely used dataset for semantic segmentation, containing 20 object categories.
  • Cityscapes: A dataset focused on urban street scenes, known for its detailed annotations.

Key Insights and Ideas

  • Direct IoU Optimization is Crucial: The paper strongly advocates for directly optimizing the evaluation metric (IoU) rather than relying on proxy losses like cross-entropy, which can lead to suboptimal results.
  • Lovász Extension as a Powerful Tool: The Lovász extension provides a principled way to derive tractable, convex surrogates for discrete, submodular loss functions like IoU, enabling their use in deep learning.
  • Per-Image vs. Dataset IoU Matters: Optimizing per-image IoU can lead to different outcomes than optimizing the global dataset IoU. The proposed "equibatch" strategy helps bridge this gap by making the training loss more representative of the dataset-level metric.
  • Small Objects Benefit Most: The Lovász-Softmax loss shows particular strength in improving the segmentation of small objects, which are often challenging for standard losses.
  • Plug-and-Play Nature: The Lovász-Softmax loss is designed to be a drop-in replacement for existing loss functions, requiring minimal changes to existing deep learning pipelines.

Key Results

  • Substantial IoU Improvement: The Lovász-Softmax loss consistently achieved higher mean IoU (mIoU) scores on both Pascal VOC and Cityscapes datasets compared to cross-entropy loss. For instance, on Pascal VOC, they reported improvements of several percentage points in mIoU.
  • Qualitative Improvements: Visualizations show that segmentations trained with the Lovász-Softmax loss are more coherent, fill gaps better, and recover small objects more effectively.
  • Competitive with State-of-the-Art: The method demonstrated state-of-the-art performance when integrated into existing deep learning architectures.
  • Efficient Computation: Despite being a more complex loss, the authors show that it can be computed efficiently, with gradients derived in polynomial time.

Limitations and Future Work

  • Computational Cost: While efficient, the Lovász extension can still be more computationally intensive than simple cross-entropy, especially for very large images or minibatches. The paper mentions that the "equibatch" strategy helps, but it's a trade-off.
  • Hyperparameter Tuning: Like any loss function, the Lovász-Softmax loss might require careful hyperparameter tuning for optimal performance.
  • Future Work: The authors suggest applying their approach to other tasks and submodular losses, and developing more specialized optimization routines that exploit the piecewise-linear nature of the Lovász extension. They also hint at exploring its application to tasks beyond segmentation.

Summary updated

Abstract

The Jaccard index, also referred to as the intersection-over-union score, is commonly employed in the evaluation of image segmentation results given its perceptual qualities, scale invariance - which lends appropriate relevance to small objects, and appropriate counting of false negatives, in comparison to per-pixel losses. We present a method for direct optimization of the mean intersection-over-union loss in neural networks, in the context of semantic image segmentation, based on the convex Lov\'asz extension of submodular losses. The loss is shown to perform better with respect to the Jaccard index measure than the traditionally used cross-entropy loss. We show quantitative and qualitative differences between optimizing the Jaccard index per image versus optimizing the Jaccard index taken over an entire dataset. We evaluate the impact of our method in a semantic segmentation pipeline and show substantially improved intersection-over-union segmentation scores on the Pascal VOC and Cityscapes datasets using state-of-the-art deep learning segmentation architectures.
BibTeX
@misc{berman2017optimization,
  author = {Maxim Berman and Matthew B. Blaschko},
  title = {{Optimization of the Jaccard index for image segmentation with the Lovász hinge}},
  year = {2017},
  eprint = {1705.08790},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1705.08790},
}
Figure