UniUSNet: A Promptable Framework for Universal Ultrasound Disease Prediction and Tissue Segmentation

Tao Tan, Zehui Lin, Zhuoneng Zhang, Xindi Hu, Zhifan Gao, Xin Yang, Yue Sun, Dong Ni, Tao Tan 0002

TL;DR

This paper introduces UniUSNet, a novel framework that unifies ultrasound image segmentation and disease classification across diverse anatomical positions and image types. By leveraging a promptable transformer architecture and a comprehensive new dataset (BroadUS-9.7K), UniUSNet achieves state-of-the-art performance and demonstrates remarkable adaptability to new tasks with minimal fine-tuning.

What Problem Does This Paper Address?

Ultrasound imaging is a cornerstone of modern diagnostics due to its accessibility and safety. However, current AI approaches for ultrasound analysis often operate in silos, focusing on either segmentation or classification for specific organs or conditions. This leads to models that are brittle, requiring extensive retraining for new tasks or datasets, and fail to capture the full diagnostic potential of ultrasound. The field lacks a universal, adaptable framework that can handle the inherent variability of ultrasound images (different anatomical positions, image qualities) and perform multiple diagnostic tasks simultaneously.

Methods

UniUSNet is built upon a Swin-Unet-like encoder-decoder architecture, enhanced with a novel promptable learning mechanism. The core innovation lies in using four types of "prompts" – representing image nature (organ/tumor), anatomical position, diagnostic task (segmentation/classification), and input type (whole/local image) – to guide the model. These prompts are embedded and injected into each transformer layer, allowing the model to dynamically adapt its processing for different scenarios without needing to redesign the network architecture. This prompt-based approach enables zero-shot and few-shot learning capabilities, making the model highly generalizable. The framework also incorporates curriculum learning, prioritizing segmentation tasks before classification during training to leverage spatial information.

Datasets

The paper introduces BroadUS-9.7K, a new, comprehensive dataset comprising over 9.7K annotations from approximately 6.9K ultrasound images across 7 distinct anatomical positions. This dataset consolidates several public datasets (BUSI, BUSIS, UDIAT, BUS-BRA, Fatty-Liver, kidneyUS, DDTI, Fetal HC, CAMUS, Appendix) and provides annotations for both segmentation and classification tasks. This unified dataset is crucial for training and validating a universal model.

Key Insights and Ideas

  • The Power of Prompts: The study strongly suggests that explicit prompts are key to achieving universal adaptability in medical imaging AI. By encoding task-specific and domain-specific information into prompts, the model can generalize far beyond its training data.
  • Unified Segmentation and Classification: UniUSNet demonstrates that it's feasible and effective to train a single model for both segmenting anatomical structures and classifying diseases, rather than relying on separate models.
  • BroadUS-9.7K as a Benchmark: The creation of this diverse, multi-task, multi-position dataset is a significant contribution, providing a much-needed benchmark for evaluating universal ultrasound AI models.
  • Efficiency in Adaptation: The prompt-based approach, combined with adapter-based fine-tuning, allows for efficient adaptation to new tasks with minimal computational cost and without catastrophic forgetting.

Key Results

  • UniUSNet achieves state-of-the-art performance on the BroadUS-9.7K dataset, outperforming single-dataset models and even SAM variants specifically adapted for ultrasound (SAMUS).
  • In segmentation tasks, UniUSNet achieved an average score of 80.01%, comparable to SAMUS (80.65%) but with significantly fewer parameters.
  • For classification tasks, UniUSNet showed strong performance, with an average of 79.67%.
  • The model demonstrated excellent generalization capabilities, with adapter-based fine-tuning outperforming scratch training on new datasets like BUSI.
  • Zero-shot inference results showed that UniUSNet significantly outperforms SAM's official weights on medical images, highlighting the importance of domain-specific adaptation.

Limitations and Future Work

  • Dataset Expansion: The authors plan to further expand the BroadUS-9.7K dataset to cover more anatomical positions and rare diseases.
  • Prompting Mechanism Refinement: Future work will focus on refining the prompting mechanism to enhance its expressiveness and efficiency.
  • Real-World Clinical Validation: While promising, the model's performance in real-world clinical settings needs further validation.
  • Computational Cost: Although more efficient than retraining, the prompt embedding and projection layers do add to the model's parameter count compared to a truly "prompt-less" model.

Summary updated

Abstract

Ultrasound is widely used in clinical practice due to its affordability, portability, and safety. However, current AI research often overlooks combined disease prediction and tissue segmentation. We propose UniUSNet, a universal framework for ultrasound image classification and segmentation. This model handles various ultrasound types, anatomical positions, and input formats, excelling in both segmentation and classification tasks. Trained on a comprehensive dataset with over 9.7K annotations from 7 distinct anatomical positions, our model matches state-of-the-art performance and surpasses single-dataset and ablated models. Zero-shot and fine-tuning experiments show strong generalization and adaptability with minimal fine-tuning. We plan to expand our dataset and refine the prompting mechanism, with model weights and code available at (https://github.com/Zehui-Lin/UniUSNet).
BibTeX
@inproceedings{tan2024uniusnet,
  author = {Tao Tan and Zehui Lin and Zhuoneng Zhang and Xindi Hu and Zhifan Gao and Xin Yang and Yue Sun and Dong Ni and Tao Tan},
  title = {{UniUSNet: A Promptable Framework for Universal Ultrasound Disease Prediction and Tissue Segmentation}},
  booktitle = {BIBM},
  year = {2024},
  doi = {10.1109/bibm62325.2024.10822429},
}
Figure