Joint Modeling of Image and Label Statistics for Enhancing Model Generalizability of Medical Image Segmentation

Shangqi Gao, Hangqi Zhou, Yibo Gao, Xiahai Zhuang

Abstract

Deep learning models for medical image segmentation often suffer from "brittleness," failing when applied to data from different scanners or sequences. We introduce BayeSeg, a novel Bayesian framework designed to enhance model generalizability. BayeSeg decomposes images into domain-invariant "contours" (structural edges) and domain-specific "basis" components (intensity/appearance). By conditioning the segmentation label solely on the structural contours, the model learns to ignore volatile intensity statistics. Our results demonstrate a significant performance leap, with BayeSeg achieving an average Dice score of 0.701 on unseen T2 MRI sequences compared to 0.118 for a standard U-Net.

Introduction

While deep learning has transformed medical imaging, its clinical utility is often limited by poor generalizability. A model trained on a specific MRI sequence, such as Late Gadolinium Enhancement (LGE), frequently fails when deployed on T2-weighted images because the intensity distributions and contrasts differ significantly, even though the underlying anatomy is identical. Manually re-labeling data for every new sequence is impractical. The challenge lies in creating a model that captures the anatomical "essence" (shape) while remaining invariant to the imaging modality's "noise" (appearance).

Methodology

The BayeSeg framework addresses this challenge through a combination of image decomposition, Bayesian statistics, and deep learning.

Image Decomposition and PGM

The core concept is to treat an input image ($y$) as a composition of two distinct parts: a contour ($x$), representing structural edges, and a basis ($n$), representing intensity and appearance. The authors formalize this using a Probabilistic Graphical Model (PGM) where the segmentation label ($z$) depends exclusively on the contour ($x$). Since anatomy is consistent across sequences while intensity varies, this decoupling fosters robustness.

The BayeSeg framework: (a) shows the Probabilistic Graphical Model (PGM) decoupling labels from intensity; (b) illustrates the deep learning implementation using ResNets for decomposition and a U-Net for segmentation.

Deep Bayesian Implementation

The framework is implemented in two stages:

  1. Decomposition Stage: Two Residual Networks (ResNets) infer the posterior distributions of the contour and the basis components.
  2. Segmentation Stage: A U-Net takes a random sample from the inferred contour distribution to generate the final segmentation label.

To handle the mathematical intractability of exact Bayesian posteriors, the authors utilize Variational Bayesian (VB) inference. They minimize a variational loss that encourages the neural networks to accurately approximate the true statistical distributions of shapes and labels.

Experimental Results

The researchers evaluated BayeSeg using the MSCMRseg (multi-sequence cardiac MR) and ACDC (cine-MRI) datasets. The primary test for generalizability involved training the model on LGE sequences and testing it on unseen T2 sequences.

Performance on Unseen Data

BayeSeg demonstrated a staggering improvement in cross-sequence generalizability. While a standard U-Net achieved a meager average Dice score of 0.118 on T2 images, BayeSeg reached 0.701. This nearly 0.6 improvement highlights the effectiveness of the contour-based decomposition.

Qualitative comparison of segmentation results on LGE (seen) and T2 (unseen) sequences. BayeSeg maintains high accuracy on T2 images where other methods, including the standard U-Net, fail significantly.

Cross-Site Stability

In cross-site evaluations (transferring from MSCMRseg to ACDC), BayeSeg exhibited the lowest performance drop, measured by the 'G' metric (G = 0.093). This indicates that the model is more robust to variations in scanner types and hospital protocols than traditional deterministic or probabilistic architectures like PU-Net.

Discussion and Conclusion

BayeSeg's success stems from the explicit modeling of labels as a function of structural contours. By forcing the segmentation network to prioritize "edges" over "pixels," the model naturally ignores the contrast shifts that typically degrade performance. Furthermore, by learning distributions rather than point estimates, the model handles ambiguity more effectively and avoids overconfident failures on out-of-distribution data.

While BayeSeg represents a significant step toward reliable, generalizable medical image segmentation, future work may address the computational overhead of the multi-network architecture and explore the framework's efficacy across different imaging modalities, such as CT to MRI transitions.

Summary updated

BibTeX
@inproceedings{gao2022joint,
  author = {Shangqi Gao and Hangqi Zhou and Yibo Gao and Xiahai Zhuang},
  title = {{Joint Modeling of Image and Label Statistics for Enhancing Model Generalizability of Medical Image Segmentation}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2022},
  doi = {10.1007/978-3-031-16443-9\_35},
}
Figure