SegHeD: Segmentation of Heterogeneous Data for Multiple Sclerosis Lesions with Anatomical Constraints

Berke Doga Basaran, Xinru Zhang, Paul M. Matthews, Wenjia Bai

Introduction

Multiple Sclerosis (MS) is a chronic inflammatory disease of the central nervous system characterized by the formation of lesions that evolve over time. Monitoring these lesions through Magnetic Resonance Imaging (MRI) is essential for assessing disease progression and treatment efficacy. While deep learning has significantly advanced automated lesion segmentation, existing models often struggle with the "real-world" heterogeneity of clinical data. Clinical datasets are frequently fragmented across sites, vary in longitudinal structure, and possess inconsistent annotation protocols—some labeling all lesions, others only new ones, and rarely accounting for vanishing lesions.

To address these challenges, we present SegHeD (Segmentation of Heterogeneous Data), a multi-task deep learning framework designed to learn from diverse, "messy" datasets by incorporating anatomical and longitudinal constraints directly into the training process.

The SegHeD Framework

SegHeD is built upon a 3D V-Net architecture featuring four specialized output heads. This multi-task design allows the model to simultaneously predict existing, new, and vanishing lesions.

Architecture and Input Handling

The model processes four primary inputs: a baseline scan ($x^{t1}$), a follow-up scan ($x^{t2}$), baseline lesion labels ($y^{t1}_a$), and a white matter mask ($x^{t2}_{wm}$). A significant strength of SegHeD is its ability to handle missing data. If a second timepoint or a baseline label is unavailable—a common occurrence in clinical practice—the framework utilizes "dummy" inputs (such as zero matrices or repeated scans) to maintain a consistent processing pipeline without sacrificing performance.

Anatomical and Longitudinal Constraints

The core innovation of SegHeD lies in its loss function, which moves beyond simple voxel-wise overlap (Dice loss) to incorporate domain-specific knowledge through three primary constraints:

  1. Longitudinal Constraints: These encode logical temporal rules. For instance, a lesion identified as "new" at $t2$ must not have been present in the $t1$ labels, yet it must be included in the total lesion count for $t2$.
  2. Volumetric Constraints: To prevent anatomically impossible predictions, the model penalizes unrealistic volume changes. Based on clinical literature, the framework allows for annual volume fluctuations between 80% and 120%; changes outside this range trigger a penalty.
  3. Spatial Constraints: Since MS lesions primarily manifest within white matter, SegHeD utilizes a white matter mask to penalize any lesion predictions occurring in non-white matter regions.

To ensure stable convergence, the authors employ a curriculum learning strategy. The model initially focuses on basic segmentation using standard Dice loss before introducing these complex anatomical constraints halfway through the training cycle.

Experimental Evaluation

The robustness of SegHeD was evaluated across five distinct MS datasets, representing a wide spectrum of clinical scenarios:

  • MS2015 (ISBI 2015): Longitudinal data with comprehensive lesion labels.
  • MS2016 (MICCAI 2016): Cross-sectional data with all-lesion labels.
  • MSSEG-2 (MICCAI 2021): Longitudinal data focusing exclusively on new lesions.
  • MSSEG-2+: An augmented version of MSSEG-2 with expert-labeled baseline data.
  • VAN: A simulated dataset designed to evaluate the detection of "vanishing" lesions (those disappearing due to repair or atrophy).

Results and Discussion

SegHeD demonstrated superior performance compared to state-of-the-art (SOTA) benchmarks, including nnU-Net and UNETR.

  • Superior Accuracy: On the MS2016 test set, SegHeD achieved a Dice score of 84.73%, representing a >10% improvement over existing SOTA methods.
  • Multi-Task Proficiency: Despite being a general-purpose model, it performed competitively with specialized models in the MSSEG-2 new-lesion challenge.
  • Vanishing Lesion Detection: The model established a significant baseline on the VAN dataset with a Dice score of 35.23%. While lower than other tasks—reflecting the difficulty of detecting subtle signal changes associated with atrophy—it represents a major step forward for a task rarely addressed in the literature.
  • Clinical Reliability: SegHeD showed the highest Pearson correlation with ground-truth lesion volumes over time, suggesting it is highly reliable for longitudinal clinical tracking.

Conclusion and Future Work

SegHeD successfully demonstrates that incorporating clinical domain knowledge as regularization can overcome the limitations of heterogeneous data. By training on multiple tasks simultaneously, the model learns the underlying relationships between different lesion types, leading to more anatomically plausible results.

Current limitations include a dependence on pre-processing tools for white matter masking and a two-timepoint window due to GPU memory constraints. Future research will focus on extending the framework to handle longer longitudinal sequences and refining the detection of vanishing lesions through more advanced temporal modeling.

Summary updated

BibTeX
@inproceedings{basaran2024seghed,
  author = {Berke Doga Basaran and Xinru Zhang and Paul M. Matthews and Wenjia Bai},
  title = {{SegHeD: Segmentation of Heterogeneous Data for Multiple Sclerosis Lesions with Anatomical Constraints}},
  booktitle = {LDTM/MMMI/ML4MHD/ML-CDS @ MICCAI},
  year = {2024},
  doi = {10.1007/978-3-031-84525-3\_5},
}
Figure