A Multi-channel Network with Image Retrieval for Accurate Brain Tissue Segmentation

Yao Sun, Yang Deng, Yue Xu, Shuo Zhang, Mingwang Zhu, Kehong YuanarXiv 2018 · arXiv

Introduction

Accurate segmentation of brain tissues—specifically cerebrospinal fluid (CSF), gray matter (GM), and white matter (WM)—is a cornerstone of neurological research and clinical diagnosis for conditions like epilepsy and Alzheimer’s. While deep learning models, particularly Convolutional Neural Networks (CNNs), have become the gold standard for this task, most models treat every image as an isolated instance. This approach fails to exploit the inherent anatomical consistency of the human brain across individuals.

This paper introduces a simple yet effective method to boost segmentation accuracy by feeding a neural network "prior knowledge" from similar cases. By using Content-Based Image Retrieval (CBIR) to find the most anatomically similar brain in a database and providing its segmentation map as an extra input channel, the researchers achieved measurable performance gains on standard benchmarks without requiring complex architectural overhauls.

Methodology

The core innovation is the addition of a fourth input channel to a standard segmentation network.

The Baseline and Oracle Experiments

The researchers utilized a multi-channel U-Net style architecture that typically takes three MRI modalities as input: T1, T2-FLAIR, and T1-IR. To test the potential benefit of spatial priors, they conducted an "oracle" experiment where the fourth channel was the image's own ground truth. This configuration achieved a near-perfect Dice ratio of 0.98, proving the network's high capacity for utilizing such information.

The CBIR Approach

In practice, ground truth is unavailable for new patients. To address this, the authors used Content-Based Image Retrieval (CBIR). For a new "query" scan, the system searches a database for the most similar existing scan. Once found, its ground truth map is registered (aligned) to the query image and fed into the network as the fourth channel. If the similarity between the query and the retrieved image falls below a specific threshold, the system reverts to the standard three-channel mode to avoid introducing noise.

The proposed workflow: using CBIR to retrieve a similar segmentation map as a fourth input channel.

Experimental Setup

The study utilized the MRBrainS13 and MRBrainS18 datasets. These provide multi-sequence 3T MRI scans with manual segmentations. The evaluation focused on CSF, GM, and WM. The training set consisted of 5 subjects, while 2 subjects were reserved for testing.

Results and Discussion

The researchers compared their proposed method against a standard three-modality baseline.

Quantitative Improvement

The method improved the average Dice ratio across all tissue types:

  • CSF: 0.8440 (Ours) vs. 0.8383 (Baseline)
  • GM: 0.8404 (Ours) vs. 0.8379 (Baseline)
  • WM: 0.8834 (Ours) vs. 0.8815 (Baseline)

While a 1% improvement may seem modest, in medical segmentation, it represents a meaningful step toward the theoretical limit established in the oracle tests.

Qualitative Analysis

Visual inspections showed that the four-channel approach produced smoother boundaries and significantly reduced random noise in continuous tissue areas compared to the three-channel version.

Visual comparison of segmentation results: (a) original T1, (b) ground truth, (c) 3-channel baseline, and (d) proposed 4-channel method.

However, some persistent issues were noted, such as parts of the skull being mistakenly classified as CSF. This is attributed to "mismatched texture" when the retrieved CBIR image is not a perfect anatomical twin.

Conclusion

The authors conclude that the potential for this method is tied directly to the quality of the retrieval. By treating a similar patient's segmentation as a feature, the network gains a "rough draft" of the expected anatomy, helping it resolve ambiguities in raw MRI signals. Future work will focus on more sophisticated CBIR techniques to find even more accurate "neighbor" images, potentially pushing performance closer to the 0.98 Dice ratio.

Summary updated

BibTeX
@misc{sun2018multi,
  author = {Yao Sun and Yang Deng and Yue Xu and Shuo Zhang and Mingwang Zhu and Kehong Yuan},
  title = {{A Multi-channel Network with Image Retrieval for Accurate Brain Tissue Segmentation}},
  year = {2018},
  eprint = {1808.00457},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1808.00457},
}
Figure