A Novel Hybrid Convolutional Neural Network for Accurate Organ Segmentation in 3D Head and Neck CT Images

Zijie Chen, Cheng Li, Junjun He, Jin Ye, Diping Song, Shanshan Wang, Lixu Gu, Yu Qiao

TL;DR

Researchers have developed OrganNet2.5D, a hybrid convolutional neural network designed to automate the tedious task of segmenting organs-at-risk (OARs) in 3D Head and Neck CT scans. By strategically mixing 2D and 3D convolutions, the model overcomes the "anisotropy" problem (where depth resolution is lower than in-plane resolution) and achieves state-of-the-art accuracy, particularly for tiny, difficult-to-detect organs like the optic chiasm.

What Problem Does This Paper Address?

In radiation therapy for head and neck cancers, doctors must precisely map out "organs-at-risk" (OARs)—like the brain stem, eyes, and optic nerves—to ensure radiation hits the tumor while sparing healthy tissue. Manually outlining these organs across hundreds of CT slices is slow, exhausting, and prone to human error.

While 3D Convolutional Neural Networks (CNNs) are the standard tool for this, they face two major hurdles:

  1. Anisotropic Resolution: In a typical CT scan, the resolution within a slice (the XY plane) is much higher than the resolution between slices (the Z-axis). Standard 3D CNNs treat all dimensions equally, which can lead to distorted feature extraction.
  2. Extreme Class Imbalance: Some organs (like the mandible) are massive, while others (like the lens of the eye) are tiny. Standard deep networks often "downsample" the image so much that these tiny organs effectively disappear from the data before the network can even process them.

Methods

The authors propose a novel architecture called OrganNet2.5D. Its name reflects its "hybrid" nature:

  • 2D Convolution Blocks: The network starts and ends with 2D convolutions. These extract sharp edge features from the high-resolution in-plane slices without being "muddied" by the lower-resolution depth information.
  • Limited Downsampling: Unlike classic architectures (like U-Net) that downsample the image four times, OrganNet2.5D only downsamples twice. This preserves the spatial detail necessary to see small organs.
  • Hybrid Dilated Convolution (HDC): To compensate for less downsampling (which usually helps a network see the "big picture"), the authors use HDC. This allows the network to have a wide "receptive field" (seeing a large area of the image at once) without losing resolution or suffering from the "gridding effect"—a common artifact where dilated convolutions skip over pixels.
  • Squeeze-and-Excitation (ResSE): The 3D blocks use residual SE units to help the network "pay attention" to the most important feature channels while suppressing noise.
  • Combined Loss Function: They use a mix of Dice Loss (to measure overlap) and Focal Loss (to force the network to focus on "hard" samples/small organs).

Datasets

The model was evaluated on two primary sources:

  1. A Large Collected Dataset: 307 3D CT samples aggregated from the Head-Neck Cetuximab collection and the Martin Vallières collection (via TCIA). This was used to segment 24 different OARs.
  2. MICCAI 2015 Challenge Dataset: A public benchmark consisting of 48 samples. This is a standard "proving ground" for head and neck segmentation, focusing on 9 OARs.

Key Insights and Ideas

  • Hybridity as Calibration: The most striking idea is using 2D layers as a "calibrator." By using 2D convolutions at the input, the model "respects" the high-resolution slices before passing the data into the 3D realm.
  • The Receptive Field Trade-off: Usually, researchers think they need many downsampling layers to understand global context. This paper argues that by using Hybrid Dilated Convolutions, you can keep the resolution high (good for small organs) while still maintaining a "global" view (good for large organs).
  • Simplicity Wins: Despite being less computationally intensive than some "Neural Architecture Search" (NAS) methods, the manual design choices in OrganNet2.5D allow it to outperform much more complex models on the smallest, most difficult organs.

Key Results

  • Small Organ Superiority: On the MICCAI 2015 dataset, OrganNet2.5D achieved the highest accuracy (66.3% Dice score) for the optic chiasm, the smallest organ in the set, outperforming much more complex models.
  • Ablation Success: The researchers proved that adding the 2D blocks specifically improved the mean Dice score from 83.9% to 84.4% on their internal dataset, confirming that "2.5D" is better than "pure 3D" for this task.
  • Competitive Performance: On the MICCAI benchmark, the model achieved a mean Dice score of 81.3%. While slightly behind some models that use massive automated architecture searches (like SCAA at 82.6%), OrganNet2.5D is significantly easier to implement and train.

Limitations and Future Work

  • Slightly Behind NAS: The model is slightly outperformed by models like SOARS and SCAA on certain large organs. These models use "Neural Architecture Search," which finds the mathematically optimal path but requires massive computing power.
  • Resolution Constraints: The inputs were resized to $256 \times 256 \times 48$. While this fits in GPU memory, some fine detail is still lost compared to the original raw CT data.
  • Future Directions: The authors suggest that integrating more advanced attention mechanisms or exploring even more sophisticated hybrid 2D/3D paths could further close the gap with automated search-based models.

Summary updated

Abstract

Radiation therapy (RT) is widely employed in the clinic for the treatment of head and neck (HaN) cancers. An essential step of RT planning is the accurate segmentation of various organs-at-risks (OARs) in HaN CT images. Nevertheless, segmenting OARs manually is time-consuming, tedious, and error-prone considering that typical HaN CT images contain tens to hundreds of slices. Automated segmentation algorithms are urgently required. Recently, convolutional neural networks (CNNs) have been extensively investigated on this task. Particularly, 3D CNNs are frequently adopted to process 3D HaN CT images. There are two issues with naïve 3D CNNs. First, the depth resolution of 3D CT images is usually several times lower than the in-plane resolution. Direct employment of 3D CNNs without distinguishing this difference can lead to the extraction of distorted image features and influence the final segmentation performance. Second, a severe class imbalance problem exists, and large organs can be orders of times larger than small organs. It is difficult to simultaneously achieve accurate segmentation for all the organs. To address these issues, we propose a novel hybrid CNN that fuses 2D and 3D convolutions to combat the different spatial resolutions and extract effective edge and semantic features from 3D HaN CT images. To accommodate large and small organs, our final model, named OrganNet2.5D, consists of only two instead of the classic four downsampling operations, and hybrid dilated convolutions are introduced to maintain the respective field. Experiments on the MICCAI 2015 challenge dataset demonstrate that OrganNet2.5D achieves promising performance compared to state-of-the-art methods.
BibTeX
@inproceedings{chen2021novel,
  author = {Zijie Chen and Cheng Li and Junjun He and Jin Ye and Diping Song and Shanshan Wang and Lixu Gu and Yu Qiao},
  title = {{A Novel Hybrid Convolutional Neural Network for Accurate Organ Segmentation in 3D Head and Neck CT Images}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2021},
  doi = {10.1007/978-3-030-87193-2\_54},
}

Loading related content...

Figure