Moving from 2D to 3D: Volumetric Medical Image Classification for Rectal Cancer Staging

Joohyung Lee, Jieun Oh, Inkyu Shin, You-sung Kim, Dae Kyung Sohn, Tae-sung Kim, In So Kweon

TL;DR

This paper introduces a novel 3D Convolutional Neural Network (CNN) architecture for automatically staging rectal cancer using volumetric MRI scans. By effectively modeling inter-slice relationships and employing a bilinear aggregation strategy, their model achieves superior performance compared to radiologists, paving the way for more accurate and efficient cancer diagnosis.

What Problem Does This Paper Address?

Accurately staging rectal cancer, particularly distinguishing between T2 and T3 stages, is crucial for determining the appropriate treatment, such as the necessity of chemotherapy and radiotherapy. While volumetric MRI scans offer rich information, precisely differentiating these stages remains a significant challenge for both radiologists and surgeons, leading to variability in diagnostic accuracy. Existing methods often rely on 2D analysis of MRI slices, requiring manual selection and potentially missing critical 3D contextual information. This paper aims to bridge this gap by developing a fully automated 3D CNN approach for volumetric rectal cancer staging.

Methods

The core of this work is a custom ResNet-based 3D CNN architecture designed to handle anisotropic medical volumes. Key methodological contributions include:

  • Late Fusion 3D Convolution: Instead of applying 3D convolutions throughout the network, which can be computationally expensive and problematic for anisotropic data, they incorporate 3D convolution only in the final layers of the encoder. This "late fusion" approach models inter-slice relationships effectively while being more efficient.
  • Bilinear Aggregation: To summarize the extracted volumetric features into a single representation for classification, the authors explore various aggregation functions. They found that bilinear encoding, a technique successful in fine-grained image recognition, performs best. This method captures richer feature interactions compared to simpler methods like average or max pooling.
  • Joint Loss Function: The model is trained using a combination of focal loss and triplet loss. Focal loss helps address class imbalance, while triplet loss aims to improve the separability between different classes (T2 and T3 stages) and the compactness within each class.

Datasets

The study utilized a retrospective dataset of 567 rectal cancer patients, comprising 168 T2-stage and 399 T3-stage MR volumes. These volumes were collected between 2004 and 2018 from seven different superconducting MRI systems (GE Healthcare and Philips Healthcare, with varying field strengths of 1.5T and 3.0T). The cancer stages were pathologically confirmed via surgical resection. The dataset is not new but is used for evaluating the proposed method.

Key Insights and Ideas

  • Late Fusion is Key for Anisotropy: The finding that late fusion of 3D information (in the final layers) is more effective than early fusion for anisotropic medical volumes is significant. This suggests a more targeted approach to leveraging 3D context without the full computational burden of a pure 3D CNN.
  • Bilinear Encoding for Depth Aggregation: Applying bilinear encoding, a technique from fine-grained recognition, to aggregate volumetric features proved highly effective. This highlights the potential for cross-domain transfer of successful deep learning techniques in medical imaging.
  • Combined Loss Function Synergy: The combination of focal loss and triplet loss demonstrated a synergistic effect, improving performance beyond using focal loss alone. This suggests that addressing both class imbalance and feature discriminability simultaneously is crucial for this task.
  • Volume-Level vs. Slice-Level Performance: The paper notes that volume-level classification is more challenging than slice-level classification, possibly due to the inclusion of more irrelevant areas in the full volume. This implies that effective aggregation and feature selection are critical for successful volume-based analysis.

Key Results

The proposed model, denoted as f-rMC5 + Bilinear + Triplet, achieved a remarkable AUC of 0.831. This performance is notably higher than the reported accuracy of professional radiologist groups (AUCs ranging from 0.64 to 0.73 in similar tasks). Specifically, the model achieved an accuracy of 79.3%, with recall rates of 62.1% for T2 and 86.7% for T3 stages. The f-rMC5 architecture, which models inter-slice relationships at the last layer, combined with bilinear aggregation and the focal+triplet loss, outperformed various other CNN architectures and aggregation functions tested.

Limitations and Future Work

  • Dataset Size and Diversity: While 567 patients is a substantial dataset, the authors acknowledge that the data was collected retrospectively and from multiple scanners. Future work could involve larger, more diverse datasets to further validate the model's robustness.
  • Generalizability: The authors believe their proposed CNN can be extended to other anisotropic VMI tasks like segmentation and detection, suggesting a broader applicability beyond rectal cancer staging.
  • Clinical Integration: While the model outperforms radiologists, further validation and integration into clinical workflows would be necessary for real-world adoption.
  • Computational Cost: Although they optimized the 3D convolution usage, 3D CNNs can still be computationally intensive. Further research into more efficient architectures or training strategies could be beneficial.

Summary updated

Abstract

Volumetric images from Magnetic Resonance Imaging (MRI) provide invaluable information in preoperative staging of rectal cancer. Above all, accurate preoperative discrimination between T2 and T3 stages is arguably both the most challenging and clinically significant task for rectal cancer treatment, as chemo-radiotherapy is usually recommended for patients with T3 (or greater) stage cancer. In this study, we present a volumetric convolutional neural network to accurately discriminate T2 from T3 stage rectal cancer with rectal MR volumes. Specifically, we propose 1) a custom ResNet-based volume encoder that models the inter-slice relationship with late fusion (i.e., 3D convolution at the last layer), 2) a bilinear computation that aggregates the resulting features from the encoder to create a volume-wise feature, and 3) a joint minimization of triplet loss and focal loss. With MR volumes of pathologically confirmed T2/T3 rectal cancer, we perform extensive experiments to compare various designs within the framework of residual learning. As a result, our network achieves an AUC of 0.831, which is higher than the reported accuracy of the professional radiologist groups. We believe this method can be extended to other volume analysis tasks.
BibTeX
@inproceedings{lee2022moving,
  author = {Joohyung Lee and Jieun Oh and Inkyu Shin and You-sung Kim and Dae Kyung Sohn and Tae-sung Kim and In So Kweon},
  title = {{Moving from 2D to 3D: Volumetric Medical Image Classification for Rectal Cancer Staging}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2022},
  url = {https://openreview.net/forum?id=776flOglU8s},
}
Figure