CNN-based Semantic Segmentation using Level Set Loss

Youngeun Kim, Seunghyeon Kim, Taekyung Kim, Changick Kim

Abstract

Convolutional Neural Networks (CNNs) are widely used in semantic segmentation. However, since CNN-based segmentation networks produce low-resolution outputs with rich semantic information, it is inevitable that spatial details (e.g., small objects and fine boundary information) of segmentation results will be lost. To address this problem, motivated by a variational approach to image segmentation (i.e., level set theory), we propose a novel loss function called the level set loss which is designed to refine spatial details of segmentation results. To deal with multiple classes in an image, we first decompose the ground truth into binary images. Note that each binary image consists of background and regions belonging to a class. Then we convert level set functions into class probability maps and calculate the energy for each class. The network is trained to minimize the weighted sum of the level set loss and the cross-entropy loss. The proposed level set loss improves the spatial details of segmentation results in a time and memory efficient way. Furthermore, our experimental results show that the proposed loss function achieves better performance than previous approaches.

Introduction

Standard CNN-based segmentation models, such as FCN or DeepLab, face a fundamental trade-off: as they go deeper to capture high-level semantic information, they lose spatial resolution due to consecutive pooling and striding layers. This results in blurry boundaries and the disappearance of small objects. While researchers often use Conditional Random Fields (CRFs) as a post-processing step to sharpen these results, CRFs are computationally expensive and not easily integrated into end-to-end training. Furthermore, the standard Cross-Entropy (CE) loss treats every pixel as an independent classification task, ignoring the spatial relationship between neighboring pixels. This research proposes a way to bake spatial awareness directly into the network's training phase using a novel Level Set Loss.

Methods

The core contribution is the adaptation of the classic Chan-Vese region-based level set method into a differentiable loss function for deep learning. The framework integrates traditional variational level set theory into the training process to incorporate spatial correlation and geometric constraints.

Overview of the proposed segmentation framework using Level Set Loss, showing multi-class decomposition and probability map conversion.
  1. Multi-class Decomposition: Since traditional level sets are binary, the multi-class ground truth is decomposed into multiple binary maps (one per class).
  2. Probability Map as Level Set: The network's output probability maps ($P_l$) are shifted from $[0, 1]$ to $[-0.5, 0.5]$ to function as the level set function $\phi$.
  3. Modified Approximated Heaviside Function (MAHF): To make the level set energy differentiable for backpropagation, a tanh-based approximation of the Heaviside function is used as a "soft" switch to determine pixel membership.
  4. Energy Minimization: The loss function calculates "energy" based on the variance of pixel intensities inside and outside predicted regions. Minimizing this energy alongside standard Cross-Entropy encourages the network to produce sharper, more spatially coherent segments.

Key Results and Insights

The authors evaluated their method on standard benchmarks including PASCAL VOC 2012, PASCAL-Context, and Cityscapes.

  • Performance Gains: On PASCAL VOC, the FCN-32s model's mean Intersection-over-Union (mIOU) jumped from 65.1% to 68.9% (+3.8%) simply by changing the loss function. DeepLab-LargeFOV saw a similar increase of nearly 4% mIOU.
  • Boundary and Small Object Refinement: The level set energy naturally penalizes jagged boundaries and provides a stronger signal for isolated clusters of pixels, helping the network recover small objects that baseline models often miss.
  • Efficiency: Training with Level Set Loss added only about 0.024 seconds per image on an NVIDIA 1080Ti, making it significantly more efficient than CRF post-processing (0.2–0.4 seconds) while providing "free" performance gains at inference time.
Qualitative comparison of segmentation results: (a) Input, (b) Ground Truth, (c) Baseline (CE loss), (d) Proposed (CE + LS loss). Note the improved boundaries and small object detection.

Conclusion

The proposed Level Set Loss successfully incorporates spatial awareness into CNN training without the computational overhead of post-processing. By treating probability maps as level set functions, the network learns to respect object boundaries and maintain spatial consistency, leading to significant improvements in both mIOU and qualitative segmentation quality across various challenging datasets.

Summary updated

Abstract

Thesedays, Convolutional Neural Networks are widely used in semantic segmentation. However, since CNN-based segmentation networks produce low-resolution outputs with rich semantic information, it is inevitable that spatial details (e.g., small bjects and fine boundary information) of segmentation results will be lost. To address this problem, motivated by a variational approach to image segmentation (i.e., level set theory), we propose a novel loss function called the level set loss which is designed to refine spatial details of segmentation results. To deal with multiple classes in an image, we first decompose the ground truth into binary images. Note that each binary image consists of background and regions belonging to a class. Then we convert level set functions into class probability maps and calculate the energy for each class. The network is trained to minimize the weighted sum of the level set loss and the cross-entropy loss. The proposed level set loss improves the spatial details of segmentation results in a time and memory efficient way. Furthermore, our experimental results show that the proposed loss function achieves better performance than previous approaches.
BibTeX
@inproceedings{kim2019cnn,
  author = {Youngeun Kim and Seunghyeon Kim and Taekyung Kim and Changick Kim},
  title = {{CNN-based Semantic Segmentation using Level Set Loss}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2019},
  url = {https://openreview.net/forum?id=xL059IEOqD2},
}
Figure