Abstract
Accurate segmentation of Multiple Sclerosis (MS) lesions is essential for disease monitoring, yet clinical practice often suffers from missing MRI modalities. While unified models like Modality Dropout (ModDrop) attempt to handle missing data, they often lack the flexibility to adapt to specific modality combinations and fail to maintain feature consistency across varying inputs. This paper presents ModDrop++, a framework that introduces a dynamic head for filter scaling and an intra-subject co-training strategy. By adaptively adjusting convolutional kernels and enforcing consistency between full and partial data views, ModDrop++ significantly improves segmentation performance, approaching the efficacy of specialized independent models.
Introduction
In an ideal clinical setting, Multiple Sclerosis (MS) diagnosis and monitoring rely on a comprehensive suite of MRI modalities, including T1-weighted (T1w), T2-weighted (T2w), and Fluid-Attenuated Inversion Recovery (FLAIR) scans. Each modality provides distinct information regarding lesion morphology and tissue characteristics. However, real-world clinical workflows are rarely ideal; scans are frequently missing due to time constraints, patient motion, or varying institutional protocols.
Traditional deep learning models for lesion segmentation are typically "all-or-nothing," requiring a fixed set of inputs to function. While the original Modality Dropout (ModDrop) technique allows a single network to handle missing modalities by randomly dropping inputs during training, it suffers from two primary limitations. First, it utilizes static parameters regardless of the input combination, limiting its representational power. Second, it fails to leverage the inherent relationship between a patient's full modality set and their partial subsets, missing an opportunity for supervised consistency.
The ModDrop++ Framework
ModDrop++ addresses these limitations through two primary architectural innovations: a dynamic filter scaling mechanism and an intra-subject co-training strategy.
1. Dynamic Head with Filter Scaling
Rather than using a static first layer, ModDrop++ employs a "dynamic head." This component receives a binary "modality code" representing the specific combination of available scans (e.g., 1011). This code is processed to generate a filter scaling matrix, which adaptively adjusts the importance of the initial layer's convolutional kernels.
This approach allows the network to specialize its feature extraction for the specific data available without the prohibitive memory cost of storing separate models for every possible combination. Notably, the authors found that scaling existing kernels is more effective than generating entire kernels from scratch, as it simplifies the optimization process and ensures faster convergence.
2. Intra-subject Co-training
To improve the robustness of the features extracted from partial data, ModDrop++ utilizes a teacher-student co-training paradigm. During the training phase, the model processes the same subject twice: once with the complete set of modalities (the "teacher" view) and once with a randomly sampled subset (the "student" view).
The framework applies a Structural Similarity Index (SSIM) loss to minimize the discrepancy between the internal feature maps of both views. This forces the network to identify and prioritize consistent anatomical features, ensuring that the absence of certain modalities does not lead the model to drastically different internal representations.
Experimental Evaluation
Datasets
The researchers validated ModDrop++ on two benchmark datasets:
- UMCL Multi-rater Consensus Dataset: Comprising 30 patients with four modalities (3D FLAIR, 2D T1w, 2D T1CE, and 2D T2w).
- ISBI 2015 Longitudinal MS Lesion Challenge: A widely used dataset featuring 21 timepoints from 5 subjects across four modalities (T1w, T2w, FLAIR, and PD).
Quantitative Results
ModDrop++ demonstrated superior performance across both datasets. On the UMCL dataset, the framework achieved a Dice Similarity Coefficient (DSC) of 0.704 in the full-modality scenario, a significant improvement over the 0.633 achieved by the standard ModDrop baseline.
Crucially, ModDrop++ maintained high performance across all 15 possible modality combinations. It consistently outperformed both the original ModDrop and a version of the model using only the dynamic head (ModDrop+), proving that the combination of adaptive filters and co-training is essential for handling missing data.
Discussion and Insights
The Role of FLAIR
The experimental results underscore the clinical importance of the FLAIR modality. In scenarios where FLAIR was missing, performance dropped across all tested models. However, ModDrop++ proved more resilient in these "worst-case" scenarios than its predecessors, effectively utilizing the remaining modalities to compensate for the missing information.
Closing the Gap
A major goal in missing-modality research is to match the performance of "Independent Models" (IM)—separate networks trained specifically for one fixed combination of inputs. While IMs are theoretically optimal, they are impractical for clinical deployment. ModDrop++ successfully narrowed the gap between a single unified model and these specialized independent networks, offering a practical solution for real-world hospital environments.
Conclusion and Future Work
ModDrop++ represents a significant step forward in robust MS lesion segmentation. By combining dynamic filter scaling with intra-subject consistency, the model achieves high accuracy even when critical MRI modalities are unavailable. While the current study is limited by the relatively small size of public MS datasets, the results suggest that the framework is highly effective. Future research will focus on validating the model on larger, more diverse clinical cohorts and exploring its applicability to other multi-modal tasks, such as brain tumor segmentation.
Summary updated