Online Reflective Learning for Robust Medical Image Segmentation

Yuhao Huang, Xin Yang, Xiaoqiong Huang, Jiamin Liang, Xinrui Zhou, Cheng Chen, Haoran Dou, Xindi Hu, Yan Cao, Dong Ni

Introduction

Deep learning models for medical image segmentation frequently encounter performance degradation when deployed across different clinical environments. This phenomenon, known as domain shift, arises from variations in scanner vendors, imaging protocols, and hospital-specific data distributions. While Domain Adaptation (DA) and Test-time Adaptation (TTA) offer potential solutions, they often struggle with the lack of ground-truth labels in target domains. This paper introduces RefSeg, a novel "reflective" learning framework that enables models to identify and correct their own segmentation errors during inference.

The RefSeg Framework

Inspired by the human "reflection-on-action" cycle, RefSeg implements an iterative loop during the testing phase to refine predictions:

  1. Action: An Attention U-Net segmentor generates a semantic heatmap from an unseen test image.
  2. Reflection: A GAN-based synthesizer uses this heatmap and an edge map of the original image to generate a "proxy" image.
  3. Evaluation: The model evaluates the structural consistency between the original input and the synthesized proxy. Discrepancies indicate segmentation flaws.
  4. Correction: The segmentor is fine-tuned using a Structural Similarity Loss based on Normalized Cross-Correlation (NCC) and Mutual Information (MI). These metrics focus on anatomical geometry rather than pixel-level style, bridging the domain gap.

The process typically converges within 10 iterations per image, allowing the model to recover from initial failures by aligning its predictions with anatomically plausible structures.

Experimental Setup

The RefSeg framework was validated across three diverse medical imaging datasets, each characterized by significant appearance shifts between vendors:

  • M&Ms Challenge: Cardiac MR images from four vendors (A, B, C, D) for multi-structure segmentation (left ventricle, myocardium, and right ventricle).
  • Developmental Dysplasia of the Hip (DDH): Ultrasound images from two vendors (E, F).
  • Fetal Head (FH): Ultrasound images from two vendors (G, H).
Visualization of the diverse datasets used, showing the significant appearance shifts between different vendors.

Results and Discussion

RefSeg demonstrated superior robustness and accuracy compared to standard baselines and existing TTA methods.

Performance and Efficiency

On the M&Ms cardiac challenge, RefSeg improved the baseline Dice score by 4.81% and significantly reduced the Hausdorff Distance. Notably, the framework is computationally efficient, requiring only 1.72 seconds per image for the 10-step adaptation—outperforming several high-ranking challenge participants in speed.

Iterative Recovery

Qualitative analysis confirms that the reflective loop effectively corrects initial segmentation errors. As the model "reflects" on the synthesized proxy, it iteratively adjusts the mask to better match the underlying anatomy.

Segmentation results for the Fetal Head (FH) task across different vendors (Ven-G and Ven-H).

Conclusion

RefSeg provides a robust mechanism for test-time adaptation in medical imaging by leveraging image synthesis as a self-supervision signal. By focusing on structural similarity rather than pixel-wise intensity, the framework effectively handles domain shifts across various modalities and vendors. Future work will explore extending this approach to 3D volumetric data and investigating more advanced similarity metrics.

Summary updated

BibTeX
@inproceedings{huang2022online,
  author = {Yuhao Huang and Xin Yang and Xiaoqiong Huang and Jiamin Liang and Xinrui Zhou and Cheng Chen and Haoran Dou and Xindi Hu and Yan Cao and Dong Ni},
  title = {{Online Reflective Learning for Robust Medical Image Segmentation}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2022},
  doi = {10.1007/978-3-031-16452-1\_62},
}

Loading related content...

Figure