Joint Modeling of Image and Label Statistics for Enhancing Model Generalizability of Medical Image Segmentation

Shangqi Gao, Hangqi Zhou, Yibo Gao, Xiahai Zhuang

TL;DR

The authors introduce BayeSeg, a deep Bayesian segmentation framework designed to solve the notorious generalizability problem in medical image AI. By mathematically decomposing an image into its domain-invariant shape (contour) and domain-specific appearance (basis), BayeSeg learns to predict anatomical labels purely from shape. This approach allows it to segment completely unseen MRI sequences and data from new clinical sites with unprecedented accuracy.

What Problem Does This Paper Address?

Deep learning models are exceptionally good at segmenting medical images—provided the test data looks exactly like the training data. However, if you train a model on cardiac MRIs from one specific scanner sequence (e.g., Late Gadolinium Enhancement, or LGE) and try to run it on another sequence (e.g., T2-weighted) or data from a different hospital, performance typically plummets.

In the real world, expert manual labeling of multi-sequence images is prohibitively expensive and time-consuming. To make medical AI viable in clinical practice, we need models that exhibit strong generalizability—the ability to perform accurately on unseen domains without requiring retraining or new manual annotations.

Methods

The core innovation is Deep Bayesian Segmentation (BayeSeg), which elegantly fuses a Probabilistic Graphical Model (PGM) with deep neural networks.

Instead of feeding a raw image directly into a segmentation network, BayeSeg breaks the image $y$ down into two distinct components:

  1. Contour ($x$): The structural boundaries and shapes, which tend to remain consistent across different imaging modalities and sites.
  2. Basis ($n$): The domain-specific intensities and appearances (modeled as a Gaussian distribution with mean $m$ and inverse variance $\rho$).

Crucially, the framework models the target segmentation label $z$ as a variable dependent only on the domain-irrelevant contour $x$.

Fig. 1. The framework of Bayesian segmentation (BayeSeg). (a) shows the probabilistic graphical model (PGM) of BayeSeg. Here, the blue circle denotes an observed image, green and orange circles denote unknown variables, and white rectangles denote hyper- parameters. (b) presents the deep learning framework of BayeSeg. Given an image, we first use ResNets to infer the posterior distributions of the contour and basis, and obtain their random samples. Then, we use U-Net to infer the posterior distribution of label, and the resulting random sample is a segmentation. Please refer to Section 2.3 for the details of network architecture and training strategy.

Because direct posterior computation is mathematically intractable, the authors employ a Variational Bayesian (VB) inference strategy implemented via neural networks:

  • Two ResNets are used during the decomposition stage to infer the variational posteriors of the contour and the basis.
  • A U-Net is then used during the segmentation stage to estimate the posterior distribution of the label based strictly on the sampled contour.

The entire system is trained end-to-end by minimizing a joint loss function consisting of standard cross-entropy ($L_{ce}$) and a specialized variational loss ($L_{var}$).

Datasets

The authors validated their approach using two prominent cardiac MRI datasets:

  • MSCMRseg (MICCAI 2019): A multi-sequence dataset consisting of 45 cardiomyopathy patient cases, each containing LGE, C0, and T2 MRI sequences. The models were trained strictly on 25 LGE images and tested on 15 T2 images to evaluate cross-sequence generalizability.
  • ACDC (MICCAI 2017): A dataset containing short-axis cardiac cine-MRIs from 100 patients. This was used entirely as an unseen test set to evaluate cross-site generalizability.

Key Insights and Ideas

  • Shape over Appearance: The fundamental "aha moment" is that while contrast and pixel intensities fluctuate wildly between MRI sequences, anatomy (shape) does not. Forcing the segmentation network to look only at the extracted contour acts as a natural regularizer against domain shifts.
  • Unsupervised Disentanglement: The network doesn't need ground-truth images of "contours" or "bases" to learn how to separate them. Instead, hierarchical Bayesian priors (like Simultaneous Autoregressive models) naturally guide the ResNets to isolate structural edges from background intensity variations.
  • Probabilistic Output: Unlike deterministic networks that output a single rigid mask, BayeSeg models the distribution of the segmentation, reducing model overconfidence—a common pitfall when deep networks encounter unfamiliar data.

Key Results

Cross-Sequence Performance (LGE $\rightarrow$ T2)

When trained on LGE and tested on the completely different T2 sequence, standard architectures failed dramatically. However, BayeSeg maintained remarkably stable performance:

Table 1. Evaluation on the task of cross-sequence cardiac segmentation. Note that all models were only trained on LGE of MSCMRseg, but tested on LGE and T2. Here, G denotes the drop of average dice, and it is used to measure model generalizability.

BayeSeg achieved an average Dice score of 0.701 on the unseen T2 sequence, outperforming standard U-Net (0.118) and a probabilistic U-Net variant (0.213) by massive margins (over 0.47 Dice). Its performance drop ($G$) was exceptionally low at just 0.107.

Fig. 2. Visualization of results on the task of cross-sequence segmentation. Here, we choose the median case of BayeSeg according to the dices of 15 LGE images.

Cross-Site Performance (MSCMRseg $\rightarrow$ ACDC)

When evaluated on the entirely separate ACDC dataset to test scanner/site variations, BayeSeg again proved superior, yielding the highest average Dice score (0.715) and the lowest performance degradation ($G = 0.093$).

Limitations and Future Work

The authors note that successful image decomposition relies heavily on selecting appropriate hyperparameters ($\gamma_\rho$ and $\gamma_v$) to properly balance the contour and basis priors. If these are poorly tuned, the adaptive balancing fails.

Future work directions were not explicitly detailed in the paper, but adjacent extensions would logically include automating hyperparameter selection and expanding this framework from 2D slices to full 3D multi-modal clinical volumes.

Summary updated

Abstract

Although supervised deep-learning has achieved promising performance in medical image segmentation, many methods cannot generalize well on unseen data, limiting their real-world applicability. To address this problem, we propose a deep learning-based Bayesian framework, which jointly models image and label statistics, utilizing the domain-irrelevant contour of a medical image for segmentation. Specifically, we first decompose an image into components of contour and basis. Then, we model the expected label as a variable only related to the contour. Finally, we develop a variational Bayesian framework to infer the posterior distributions of these variables, including the contour, the basis, and the label. The framework is implemented with neural networks, thus is referred to as deep Bayesian segmentation. Results on the task of cross-sequence cardiac MRI segmentation show that our method set a new state of the art for model generalizability. Particularly, the BayeSeg model trained with LGE MRI generalized well on T2 images and outperformed other models with great margins, i.e., over 0.47 in terms of average Dice. Our code is available at https://zmiclab.github.io/projects.html.
BibTeX
@inproceedings{gao2022joint,
  author = {Shangqi Gao and Hangqi Zhou and Yibo Gao and Xiahai Zhuang},
  title = {{Joint Modeling of Image and Label Statistics for Enhancing Model Generalizability of Medical Image Segmentation}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2022},
  doi = {10.1007/978-3-031-16443-9\_35},
}
Figure