Robust Segmentation of Brain MRI in the Wild with Hierarchical CNNs and No Retraining

Benjamin Billot, Colin Magdamo, Steven E. Arnold, Sudeshna Das, Juan Eugenio IglesiasLecture Notes in Computer Science 2022 · DOI · OpenAlex

Abstract

Retrospective analysis of brain MRI scans acquired in the clinic has the potential to enable neuroimaging studies with sample sizes much larger than those found in research datasets. However, analysing such clinical images "in the wild" is challenging, since subjects are scanned with highly variable protocols (MR contrast, resolution, orientation, etc.). Nevertheless, recent advances in convolutional neural networks (CNNs) and domain randomisation for image segmentation, best represented by the publicly available method SynthSeg, may enable morphometry of clinical MRI at scale. In this work, we first evaluate SynthSeg on an uncurated, heterogeneous dataset of more than 10,000 scans acquired at Massachusetts General Hospital. We show that SynthSeg is generally robust, but frequently falters on scans with low signal-to-noise ratio or poor tissue contrast. Next, we propose SynthSeg+, a novel method that greatly mitigates these problems using a hierarchy of conditional segmentation and denoising CNNs. We show that this method is considerably more robust than SynthSeg, while also outperforming cascaded networks and state-of-the-art segmentation denoising methods. Finally, we apply our approach to a proof-of-concept volumetric study of ageing, where it closely replicates atrophy patterns observed in research studies conducted on high-quality, 1mm, T1-weighted scans.

Introduction

Most brain segmentation software is "brittle," typically optimized for 1mm isotropic T1-weighted scans used in research. However, clinical datasets are a "wild west" of imaging, characterized by thick slices (low resolution), varying orientations, different contrasts (T2, FLAIR), and significant noise. Standard Convolutional Neural Networks (CNNs) often fail when encountering data that deviates from their training set. While the original SynthSeg addressed this through domain randomization—training on synthetic images—it still struggled with scans exhibiting very low signal-to-noise ratios (SNR) or poor tissue contrast. This paper introduces SynthSeg+, a hierarchical framework designed to bridge this gap and unlock the potential of millions of clinical scans for large-scale studies without requiring retraining.

Methods

The SynthSeg+ framework utilizes a three-stage "coarse-to-fine" hierarchical architecture to handle highly variable clinical brain MRIs.

Hierarchical Architecture

The system consists of three primary modules:

  1. Segmenter $S_1$ (Coarse): Performs a robust initial task of grouping voxels into four broad tissue classes (white matter, grey matter, CSF, and cerebellum).
  2. Denoiser $D$: A network that "cleans" the output of $S_1$. Crucially, it only processes labels, forcing it to learn high-level anatomical priors to correct semantic errors.
  3. Segmenter $S_2$ (Fine): Combines the original image with the refined tissue maps from $D$ to produce a detailed 44-region segmentation.
The SynthSeg+ hierarchical pipeline: from input image to coarse tissue classes, anatomical denoising, and final fine-grained segmentation.

Training via Domain Randomization

The entire system is trained using domain randomization. The networks never see a "real" MRI during training. Instead, they are trained on label maps that have been randomly deformed and "filled" with random intensities, noise, and artifacts. This approach makes the model agnostic to specific MRI contrasts or resolutions at test time.

Illustration of the synthetic training data generation process, showing label deformation and the simulation of various imaging artifacts.

Datasets

Training was conducted using label maps from 1,020 high-quality 1mm T1 scans. Evaluation was performed on a massive, uncurated dataset of 10,520 clinical scans from Massachusetts General Hospital (MGH), covering a wide range of protocols. Validation utilized 520 "silver standard" scans with labels propagated from high-quality T1 scans.

Results and Discussion

SynthSeg+ demonstrates significant improvements in robustness and accuracy over previous methods.

  • Robustness Gains: On scans where the original SynthSeg failed significantly ("Big Fails"), SynthSeg+ improved the Dice score by 23.5 points.
  • Superior Performance: The method outperformed standard denoising techniques and cascaded networks by at least 3 Dice points and 0.35 mm in Mean Surface Distance.
  • Real-World Application: In a study of 10,000 scans, SynthSeg+ successfully replicated aging-related atrophy patterns, such as hippocampal shrinkage, even on scans with 6.5mm slice spacing.

The core strength of SynthSeg+ lies in using anatomy as a prior. By employing a denoiser that only sees labels, the model incorporates "anatomical common sense," allowing it to correct errors caused by noise. Furthermore, the hierarchical approach simplifies the learning task for each CNN, enhancing generalization on edge-case scans.

Conclusion

SynthSeg+ provides a "plug-and-play" tool for robust brain MRI segmentation in clinical settings. While the denoising step can occasionally over-smooth complex boundaries or introduce theoretical anatomical bias, its ability to handle "in the wild" data without retraining is a major advancement. Future work may extend this hierarchical approach to lesion segmentation or longitudinal disease tracking.

Code and models are available at: https://github.com/BBillot/SynthSeg

Summary updated

BibTeX
@inproceedings{billot2022robust,
  author = {Benjamin Billot and Colin Magdamo and Steven E. Arnold and Sudeshna Das and Juan Eugenio Iglesias},
  title = {{Robust Segmentation of Brain MRI in the Wild with Hierarchical CNNs and No Retraining}},
  booktitle = {Lecture Notes in Computer Science},
  year = {2022},
  doi = {10.1007/978-3-031-16443-9\_52},
}

Loading related content...

Figure