Generalize Ultrasound Image Segmentation via Instant and Plug & Play Style Transfer

Zhendong Liu, Xiaoqiong Huang, Xin Yang, Rui Gao, Rui Li, Yuanji Zhang, Yankai Huang, Guangquan Zhou, Yi Xiong, Alejandro F. Frangi, Dong Ni

Abstract

Deep segmentation models that generalize to images with unknown appearance are important for real-world medical image analysis. Retraining models leads to high latency and complex pipelines, which are impractical in clinical settings. The situation becomes more severe for ultrasound image analysis because of their large appearance shifts. In this paper, we propose a novel method for robust segmentation under unknown appearance shifts. Our contribution is three-fold. First, we advance a one-stage plug-and-play solution by embedding hierarchical style transfer units into a segmentation architecture. Our solution can remove appearance shifts and perform segmentation simultaneously. Second, we adopt Dynamic Instance Normalization to conduct precise and dynamic style transfer in a learnable manner, rather than previously fixed style normalization. Third, our solution is fast and lightweight for routine clinical adoption. Given 400*400 image input, our solution only needs an additional 0.2ms and 1.92M FLOPs to handle appearance shifts compared to the baseline pipeline. Extensive experiments are conducted on a large dataset from three vendors demonstrate our proposed method enhances the robustness of deep segmentation models.

Introduction

Deep learning has revolutionized medical imaging, yet models often suffer from significant performance degradation when applied to data from unseen domains. In ultrasound imaging, "appearance shifts"—variations in graininess, contrast, and speckle patterns caused by different hardware vendors or clinical protocols—pose a major hurdle. A model trained on Vendor A images often fails on Vendor B. Traditional solutions like Domain Adaptation (DA) require retraining on target datasets, which is clinically impractical. This paper introduces DINSeg, a one-stage, "plug-and-play" framework designed to handle these shifts instantly.

Methodology: DINSeg and Dynamic Instance Normalization

The core innovation of DINSeg is the integration of style transfer directly into a standard segmentation architecture (e.g., U-Net). Unlike previous two-stage methods that stylize then segment, DINSeg performs both simultaneously.

Dynamic Instance Normalization (DIN)

The framework utilizes Dynamic Instance Normalization (DIN) instead of standard Adaptive Instance Normalization (AdaIN). While AdaIN uses fixed statistics (mean and standard deviation), DIN employs learnable "Weight Nets" and "Bias Nets" to generate precise affine parameters. This allows the model to capture and normalize sophisticated, non-linear style patterns characteristic of ultrasound images.

Plug-and-Play Integration

DIN units are "plugged" into the encoder and skip connections of a pre-trained segmentation model. During training, the segmentation backbone is frozen, and only the DIN units are trained. They learn to map any incoming image appearance back to the "style" of the original training domain (Vendor A) using a large natural image dataset (MS-COCO) to provide content variety.

The DINSeg architecture showing the integration of DIN units into the testing pipeline.

Experimental Results

The authors evaluated DINSeg on a dataset of 6,532 ultrasound images of the fetal head (FH) and fetal abdomen (FA) from three vendors (Ven-A, Ven-B, Ven-C). Vendor A served as the source domain.

Performance and Efficiency

The multi-unit variant, M-DINSeg, significantly outperformed baseline models on unseen vendors. On Vendor-B, it improved the Dice score by approximately 5% for fetal abdomen and 2% for fetal head. Crucially, this robustness comes with minimal computational overhead. The style transfer component adds as little as 0.2ms of latency and 1.92M FLOPs, making it ideal for real-time clinical applications.

Qualitative Analysis

Visual comparisons demonstrate that DINSeg effectively recovers segmentation masks that were otherwise fragmented or incorrect in the baseline model when faced with vendor-specific appearance shifts. Heatmaps further illustrate how the model focuses more accurately on the target anatomy after style normalization.

Qualitative comparison of segmentation results across different vendors and organs, showing the improvement from baseline to DINSeg.

Conclusion

DINSeg offers a lightweight and efficient solution for generalizing ultrasound segmentation models across different vendors. By treating domain generalization as a real-time style transfer problem and utilizing learnable normalization, the method achieves high accuracy without the need for target-domain retraining. Future work may explore extending this "plug-and-play" approach to other medical imaging modalities like MRI and CT where vendor variations are also prevalent.

Summary updated

Abstract

Deep segmentation models that generalize to images with unknown appearance are important for real-world medical image analysis. Retraining models leads to high latency and complex pipelines, which are impractical in clinical settings. The situation becomes more severe for ultrasound image analysis because of their large appearance shifts. In this paper, we propose a novel method for robust segmentation under unknown appearance shifts. Our contribution is three-fold. First, we advance a one-stage plug-and-play solution by embedding hierarchical style transfer units into a segmentation architecture. Our solution can remove appearance shifts and perform segmentation simultaneously. Second, we adopt Dynamic Instance Normalization to conduct precise and dynamic style transfer in a learnable manner, rather than previously fixed style normalization. Third, our solution is fast and lightweight for routine clinical adoption. Given 400*400 image input, our solution only needs an additional 0.2ms and 1.92M FLOPs to handle appearance shifts compared to the baseline pipeline. Extensive experiments are conducted on a large dataset from three vendors demonstrate our proposed method enhances the robustness of deep segmentation models.
BibTeX
@misc{liu2021generalize,
  author = {Zhendong Liu and Xiaoqiong Huang and Xin Yang and Rui Gao and Rui Li and Yuanji Zhang and Yankai Huang and Guangquan Zhou and Yi Xiong and Alejandro F. Frangi and Dong Ni},
  title = {{Generalize Ultrasound Image Segmentation via Instant and Plug \& Play Style Transfer}},
  year = {2021},
  eprint = {2101.03711},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2101.03711},
}
Figure