Hybrid Graph Convolutional Neural Networks for Landmark-Based Anatomical Segmentation

Nicolás Gaggion, Lucas Mansilla, Diego H. Milone, Enzo Ferrante

Abstract

In this work, we address the problem of landmark-based segmentation for anatomical structures. We propose HybridGNet, an encoder-decoder neural architecture which combines standard convolutions for image feature encoding with graph convolutional neural networks to decode plausible representations of anatomical structures. We benchmark the proposed architecture against other standard landmark and pixel-based models for anatomical segmentation in chest X-ray images and find that HybridGNet is more robust to image occlusions. We also show that it can be used to construct landmark-based segmentations from pixel-level annotations. Our experimental results suggest that HybridGNet produces accurate and anatomically plausible landmark-based segmentations by naturally incorporating shape constraints within the decoding process via spectral convolutions.

Introduction

In medical imaging, segmentation typically involves assigning labels to every pixel in an image. While standard architectures like the U-Net excel at this task, they often lack an inherent understanding of global anatomical structure. Consequently, pixel-based models may predict anatomically impossible shapes, such as organs with holes or jagged edges, because they lack topological constraints.

Anatomical structures like lungs, hearts, and bones follow regular, predictable patterns. Traditionally, researchers used Point Distribution Models (PDMs) to capture this, but these models often rely on linear assumptions (like PCA) that struggle with complex variations. HybridGNet bridges this gap by combining the feature-learning power of deep learning with the structural constraints of graph-based representations, ensuring that the output is always a topologically "legal" anatomical shape.

Methods: The HybridGNet Architecture

HybridGNet is an encoder-decoder architecture that operates across two domains: image features and graph representations.

  1. The Image Encoder: A standard CNN encoder using residual blocks processes a chest X-ray to produce a low-dimensional latent vector representing the image's essential features.
  2. The Graph Decoder: Instead of a deconvolutional decoder that generates pixels, HybridGNet utilizes a Graph Convolutional Neural Network (GCNN). This decoder takes the latent vector and predicts the spatial coordinates of a set of landmarks (nodes) connected in a predefined graph.
The HybridGNet architecture combines a convolutional image encoder with a graph convolutional decoder to map images directly to anatomical landmarks.

Technical Innovations

  • Spectral Graph Convolutions: The model employs Chebyshev convolutions, allowing it to perform operations on the graph using the graph Laplacian. This enables the network to learn spatial relationships between neighboring landmarks.
  • VAE Pre-training: To stabilize training, the authors first trained separate Variational Autoencoders (VAEs) for images and landmark shapes, then "stitched" the image encoder to the graph decoder for fine-tuning.
  • Dual-Branch Variants: The study also explored "Dual" versions that output both a graph and a standard pixel mask, utilizing skip connections to share information between tasks.

Experimental Setup and Results

The model was evaluated using the JSRT (Japanese Society of Radiological Technology) Database, containing 247 high-resolution chest X-rays with 166 landmarks per image. The dataset was split into training (70%), validation (10%), and testing (20%) sets using 3-fold cross-validation.

Quantitative Performance

HybridGNet outperformed standard baselines, including PCA-based models and fully connected VAEs, across key metrics. Specifically, it achieved lower Mean Squared Error (MSE) and Hausdorff Distance (HD), indicating that its predicted contours were closer to the ground truth.

Quantitative comparison of HybridGNet variants against baselines (VAE, PCA, MultiAtlas) using MSE and Hausdorff Distance metrics.

Key Insights

  • Topology as a Prior: By using a graph with fixed connectivity, the model cannot produce a topologically broken shape, naturally incorporating anatomical constraints.
  • Robustness to Occlusion: HybridGNet demonstrated superior stability during "stress tests" where portions of the X-ray were obscured. Because it maps the entire image to a global latent space, it can infer missing boundaries based on the surrounding visible anatomy.
  • Anatomical Plausibility: Qualitative analysis showed that HybridGNet produced smoother and more realistic shapes than baseline models, particularly for complex structures like the clavicles.

Conclusion

HybridGNet represents a significant step forward in anatomical segmentation by integrating graph-based priors into deep learning architectures. By treating anatomy as a graph rather than a collection of independent pixels, the model achieves greater robustness and plausibility. Future work may explore dynamic graph connectivity and extending the architecture to 3D medical imaging volumes.

Summary updated

Abstract

In this work we address the problem of landmark-based segmentation for anatomical structures. We propose HybridGNet, an encoder-decoder neural architecture which combines standard convolutions for image feature encoding, with graph convolutional neural networks to decode plausible representations of anatomical structures. We benchmark the proposed architecture considering other standard landmark and pixel-based models for anatomical segmentation in chest x-ray images, and found that HybridGNet is more robust to image occlusions. We also show that it can be used to construct landmark-based segmentations from pixel level annotations. Our experimental results suggest that Hybrid-Net produces accurate and anatomically plausible landmark-based segmentations, by naturally incorporating shape constraints within the decoding process via spectral convolutions.
BibTeX
@inproceedings{gaggion2021hybrid,
  author = {Nicolás Gaggion and Lucas Mansilla and Diego H. Milone and Enzo Ferrante},
  title = {{Hybrid Graph Convolutional Neural Networks for Landmark-Based Anatomical Segmentation}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2021},
  url = {https://openreview.net/forum?id=S3ZdToCUvfz},
}
Figure