How GNNs Facilitate CNNs in Mining Geometric Information from Large-Scale Medical Images

Yiqing Shen, Bingxin Zhou, Xinye Xiong, Ruitian Gao, Yu Guang Wang

Introduction

Digital pathology relies on "gigapixel" whole-slide images (WSIs) that contain vast amounts of information. While Convolutional Neural Networks (CNNs) have become the standard for analyzing these images, they are primarily optimized for identifying textures and pixel-level patterns—morphology. However, tissue is more than just a collection of textures; it is a complex map of cellular "neighborhoods." Standard CNNs struggle to capture the topological structure of tissue, such as how immune cells cluster around a tumor, which are critical indicators of cancer progression.

This research introduces a fusion framework that combines CNNs with Graph Neural Networks (GNNs). By treating the arrangement of cells as a graph, the model mines geometric information that standard grid-based CNNs typically ignore, leading to significant improvements in biomarker prediction.

Overview of the hierarchical diagnostic framework across patient, WSI, and patch levels.

Methodology: The Dual-Stream Fusion Framework

The proposed framework processes histology patches in parallel through two distinct streams:

1. The CNN Stream

This stream extracts global image-level features, focusing on morphology (textures and shapes). It utilizes standard backbones such as ResNet, DenseNet, or MobileNetV3 to generate image embeddings for each tissue patch.

2. The GNN Stream

The GNN stream is designed to capture the "geometric signature" of cell neighborhoods. The process involves:

  • Nuclei Segmentation: Identifying individual cell nuclei using a specialized network (CA2.5-Net).
  • Graph Construction: Building a "Cell Graph" where each nucleus serves as a node. Nodes are assigned 94 "pathomics" features (color, shape, texture), and edges are drawn based on spatial proximity.
  • Feature Extraction: A GNN (GCN or GIN) processes this graph to distill local geometric interactions.
From raw histology patch to cell graph: segmentation, overlay, and final graph representation.

3. Fusion and Prediction

The core innovation lies in the fusion layer, which integrates the CNN and GNN outputs. The authors evaluated two strategies: a simple Multi-Layer Perceptron (MLP) for feature concatenation and a Transformer-based approach using self-attention to adaptively align the two information types. Final patient-level diagnosis is reached via a majority vote of patch-level predictions.

Key Insights and Results

The framework was evaluated on three histology benchmarks (CRC-MSI, STAD-MSI, and GIST-PDL1) and a synthetic dataset.

  • Complementary Features: Synthetic experiments confirmed that geometric (GNN) and morphological (CNN) features are non-redundant. As tasks become more geometric, the GNN's importance grows, but the fusion model consistently outperforms either stream alone.
  • Performance Gains: The integrated models achieved a consistent AUC improvement of over 5% compared to standalone CNNs or GNNs. At the patient level, some configurations saw diagnostic accuracy boosts of up to 23%.
  • Transformer Superiority: The Transformer-based fusion emerged as the most effective method for aligning abstract features from different architectures.
  • Backbone Agnostic: The performance boost remained consistent across various CNN backbones, indicating broad applicability.

Conclusion

By representing tissue as a "social network" of cells, this fusion framework overcomes the limitations of traditional CNNs in digital pathology. While the pipeline currently requires significant preprocessing for segmentation and graph construction, it sets a new state-of-the-art for biomarker prediction. Future work aims to automate graph generation directly from raw pixels, potentially uncovering even more complex latent spatial relationships within the tumor microenvironment.

Summary updated

BibTeX
@inproceedings{shen2023how,
  author = {Yiqing Shen and Bingxin Zhou and Xinye Xiong and Ruitian Gao and Yu Guang Wang},
  title = {{How GNNs Facilitate CNNs in Mining Geometric Information from Large-Scale Medical Images}},
  booktitle = {BIBM},
  year = {2023},
  doi = {10.1109/bibm58861.2023.10385379},
}
Figure