Improving anatomical plausibility in medical image segmentation via hybrid graph neural networks: applications to chest x-ray analysis

Nicolás Gaggion, Lucas Mansilla, Candelaria Mosquera, Diego H. Milone, Enzo Ferrante

Introduction

Medical image segmentation is a cornerstone of computer-aided diagnosis. While state-of-the-art deep learning models like U-Net achieve high pixel-level accuracy, they often fail to maintain "anatomical plausibility." Because standard loss functions treat pixels as largely independent, models can produce fragmented organs or holes—results that are biologically impossible. This paper introduces HybridGNet, an architecture designed to enforce structural constraints by combining Convolutional Neural Networks (CNNs) with Graph Neural Networks (GNNs).

Methodology: The HybridGNet Architecture

HybridGNet moves away from pixel-level masks, instead representing anatomical structures as graphs where nodes correspond to landmarks. The architecture follows an encoder-decoder structure:

  1. CNN Encoder: A residual CNN extracts high-level features from the input image.
  2. GCNN Decoder: A Graph Convolutional Neural Network (GCNN) uses spectral convolutions to learn spatial relationships between landmarks.
  3. Image-to-Graph Skip Connections (IGSC): This novel component bridges the gap between image features and graph nodes. Using RoIAlign, the model samples local image features based on current node positions, allowing the decoder to refine landmark locations using local texture.
HybridGNet architecture overview showing the CNN encoder, GCNN decoder, and the Image-to-Graph Skip Connection (IGSC) mechanism.

Experimental Setup

The model was evaluated on several chest X-ray (CXR) datasets:

  • JSRT: Primary training set with 120 expert landmarks for heart and lungs.
  • Montgomery & Shenzhen: External test sets for domain shift evaluation.
  • Padchest: Used for testing robustness against clinical occlusions (e.g., pacemakers) and cardiothoracic ratio (CTR) estimation.

Results and Discussion

HybridGNet demonstrates significant improvements in both quantitative metrics and qualitative plausibility. Unlike pixel-based methods, the graph-based output ensures a continuous, single-component topology by construction.

Quantitative performance table and qualitative comparison of segmentation results across various landmark-based and pixel-level methods.

Key Findings

  • Robustness to Occlusions: In cases where images contain "black boxes" or pacemakers, HybridGNet "hallucinates" missing boundaries by relying on its learned anatomical prior, whereas pixel-based models like nnU-Net often produce holes.
  • Efficiency: HybridGNet is computationally efficient, with an inference time of 0.53s per image, roughly five times faster than nnU-Net.
  • Demographic Bias: A critical observation was the performance drop in pediatric cases. Since the training data lacked children, the model struggled with their different anatomical proportions, highlighting the need for age-diverse training sets.

Conclusion

HybridGNet successfully integrates global anatomical priors with local image features to produce highly plausible segmentations. By guaranteeing topological correctness and offering superior robustness to occlusions, it provides a more reliable tool for clinical tasks like CTR estimation. Future work will explore extending this framework to 3D medical imaging and addressing demographic gaps in training data.

Summary updated

BibTeX
@article{gaggion2023improving,
  author = {Nicolás Gaggion and Lucas Mansilla and Candelaria Mosquera and Diego H. Milone and Enzo Ferrante},
  title = {{Improving anatomical plausibility in medical image segmentation via hybrid graph neural networks: applications to chest x-ray analysis}},
  journal = {IEEE TMI},
  year = {2023},
  doi = {10.1109/tmi.2022.3224660},
}
Figure