Interpretable Small Training Set Image Segmentation Network Originated from Multi-Grid Variational Model

Junying Meng, Weihong Guo, Jun Liu, Mingrui Yang

TL;DR

This paper introduces MSNet, a segmentation network that bridges the gap between classical variational models and deep learning. By "unrolling" a multi-grid solver for the Mumford-Shah model, the authors create a highly interpretable, parameter-efficient architecture that outperforms state-of-the-art models like U-Net, especially when training data is extremely limited.

What Problem Does This Paper Address?

In medical imaging, segmenting structures (like tumors or muscles) is critical but difficult. Currently, two worlds exist:

  1. Deep Learning (DL): Highly accurate but acts as a "black box." It requires massive amounts of manually labeled data, which is expensive and rare in clinical settings.
  2. Variational Models: Classical mathematical approaches (like the Mumford-Shah model) that are fully interpretable and work with little data. However, they rely on "hand-crafted" rules that don't adapt well to complex textures and require tedious parameter tuning.

The authors tackle the "black box" problem and the "data hunger" of DL by creating a network that is mathematically derived from a classical model but uses learning to handle the complex parts.

Methods

The core of the paper is a technique called Algorithm Unrolling. Instead of designing a neural network by stacking random layers, the authors take an iterative mathematical algorithm and turn each iteration into a network layer.

  • The Model: They start with the Mumford-Shah (MS) model, which seeks a piece-wise smooth approximation of an image.
  • Learnable Regularity: In classical MS, the "smoothing" rules are fixed. Here, they replace them with a data-adaptive, learnable operator ($\mathcal{L}$). This allows the network to learn what "smoothness" looks like for a specific dataset (e.g., muscle vs. blood cells).
  • The Multi-Grid Framework: To solve the MS equations efficiently, they use a multi-grid solver. This involves solving the problem at a fine scale, moving to a coarse scale to capture global features, and then refining back to fine.
  • U-Type Architecture: Unrolling this multi-grid solver naturally results in a U-shaped network. This provides a rigorous mathematical justification for why the popular U-Net architecture is so effective for segmentation: it is essentially a multi-scale numerical solver.

Datasets

The authors evaluated their model on three distinct tasks to prove generalizability:

  1. REFUGE Dataset: Fundus images for segmenting the optic disc and cup (crucial for glaucoma diagnosis).
  2. White Blood Cell (WBC) Dataset: Microscopic images of blood cells.
  3. 3D Thigh Muscle MR Images: A complex 3D task involving segmenting multiple tightly packed muscle groups with low contrast.

Key Insights and Ideas

  • Mathematical "Aha!": The paper demonstrates that a U-Net-like structure isn't just a lucky design—it is the neural network equivalent of a multi-grid numerical algorithm.
  • Parameter Efficiency: Because the network's structure is guided by a mathematical model, it doesn't need millions of "blind" parameters. The 2D MSNet uses only 111,456 parameters, compared to over 31 million in a standard U-Net.
  • Inductive Bias: By originating from a variational model, the network has a strong "inductive bias" (prior knowledge). This allows it to maintain high performance even when the training set is shrunk to just 20 images.
  • Interpretability: Every part of the network corresponds to a step in an energy minimization problem, making it easier for researchers to understand how the network is arriving at its segmentation.

Key Results

MSNet consistently outperformed U-Net, DeepLabV3+, and R2U-Net across all metrics (Accuracy, mIoU, and Dice Coefficient).

  • Small Data Performance: On the REFUGE dataset with only 20 training images, MSNet achieved a Dice score of 0.8147, significantly higher than U-Net (0.7416).
  • 3D Segmentation: In the 3D thigh muscle task, MSNet achieved a Dice score of 0.9185 with only 4 training volumes, while 3D U-Net trailed at 0.8560.
  • Qualitative Quality: Visual results show that MSNet produces smoother, more continuous boundaries and is much better at distinguishing between adjacent objects with similar intensities (like different thigh muscles).

Limitations and Future Work

  • Mathematical Complexity: While the resulting network is efficient, the derivation requires a deep understanding of variational calculus and numerical methods, which might be a barrier for some practitioners.
  • Operator Design: The learnable operator $\mathcal{L}$ is currently a simple combination of convolutions and ReLUs. The authors suggest that exploring more complex architectures for this specific component could further boost performance.
  • Generalization: While tested on three datasets, further testing on more diverse 3D medical modalities (like CT or Ultrasound) would strengthen the case for its clinical adoption.

Summary updated

Abstract

The main objective of image segmentation is to divide an image into homogeneous regions for further analysis. This is a significant and crucial task in many applications such as medical imaging. Deep learning (DL) methods have been proposed and widely used for image segmentation. However, these methods usually require a large amount of manually segmented data as training data and suffer from poor interpretability (known as the black box problem). The classical Mumford-Shah (MS) model is effective for segmentation and provides a piece-wise smooth approximation of the original image. In this paper, we replace the hand-crafted regularity term in the MS model with a data adaptive generalized learnable regularity term and use a multi-grid framework to unroll the MS model and obtain a variational model-based segmentation network with better generalizability and interpretability. This approach allows for the incorporation of learnable prior information into the network structure design. Moreover, the multi-grid framework enables multi-scale feature extraction and offers a mathematical explanation for the effectiveness of the U-shaped network structure in producing good image segmentation results. Due to the proposed network originates from a variational model, it can also handle small training sizes. Our experiments on the REFUGE dataset, the White Blood Cell image dataset, and 3D thigh muscle magnetic resonance (MR) images demonstrate that even with smaller training datasets, our method yields better segmentation results compared to related state of the art segmentation methods.
BibTeX
@misc{meng2023interpretable,
  author = {Junying Meng and Weihong Guo and Jun Liu and Mingrui Yang},
  title = {{Interpretable Small Training Set Image Segmentation Network Originated from Multi-Grid Variational Model}},
  year = {2023},
  eprint = {2306.14097},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2306.14097},
}
Figure