Introduction
Deep learning models for medical image segmentation frequently encounter performance degradation when deployed across different clinical environments. This phenomenon, known as domain shift, arises from variations in scanner vendors, imaging protocols, and hospital-specific data distributions. While Domain Adaptation (DA) and Test-time Adaptation (TTA) offer potential solutions, they often struggle with the lack of ground-truth labels in target domains. This paper introduces RefSeg, a novel "reflective" learning framework that enables models to identify and correct their own segmentation errors during inference.
The RefSeg Framework
Inspired by the human "reflection-on-action" cycle, RefSeg implements an iterative loop during the testing phase to refine predictions:
- Action: An Attention U-Net segmentor generates a semantic heatmap from an unseen test image.
- Reflection: A GAN-based synthesizer uses this heatmap and an edge map of the original image to generate a "proxy" image.
- Evaluation: The model evaluates the structural consistency between the original input and the synthesized proxy. Discrepancies indicate segmentation flaws.
- Correction: The segmentor is fine-tuned using a Structural Similarity Loss based on Normalized Cross-Correlation (NCC) and Mutual Information (MI). These metrics focus on anatomical geometry rather than pixel-level style, bridging the domain gap.
The process typically converges within 10 iterations per image, allowing the model to recover from initial failures by aligning its predictions with anatomically plausible structures.
Experimental Setup
The RefSeg framework was validated across three diverse medical imaging datasets, each characterized by significant appearance shifts between vendors:
- M&Ms Challenge: Cardiac MR images from four vendors (A, B, C, D) for multi-structure segmentation (left ventricle, myocardium, and right ventricle).
- Developmental Dysplasia of the Hip (DDH): Ultrasound images from two vendors (E, F).
- Fetal Head (FH): Ultrasound images from two vendors (G, H).
Results and Discussion
RefSeg demonstrated superior robustness and accuracy compared to standard baselines and existing TTA methods.
Performance and Efficiency
On the M&Ms cardiac challenge, RefSeg improved the baseline Dice score by 4.81% and significantly reduced the Hausdorff Distance. Notably, the framework is computationally efficient, requiring only 1.72 seconds per image for the 10-step adaptation—outperforming several high-ranking challenge participants in speed.
Iterative Recovery
Qualitative analysis confirms that the reflective loop effectively corrects initial segmentation errors. As the model "reflects" on the synthesized proxy, it iteratively adjusts the mask to better match the underlying anatomy.
Conclusion
RefSeg provides a robust mechanism for test-time adaptation in medical imaging by leveraging image synthesis as a self-supervision signal. By focusing on structural similarity rather than pixel-wise intensity, the framework effectively handles domain shifts across various modalities and vendors. Future work will explore extending this approach to 3D volumetric data and investigating more advanced similarity metrics.
Summary updated