Comprehensive Multimodal Segmentation in Medical Imaging: Combining YOLOv8 with SAM and HQ-SAM Models

Sumit Pandey, Kuan-Fu Chen, Erik B. DamWorkshops @ ICCV 2023 · DOI

Abstract

Medical image segmentation is vital for clinical diagnosis but traditionally demands extensive manual labeling or large-scale specialized training. This paper introduces an automated pipeline that bridges the gap between efficient object detection and powerful foundation models. By using YOLOv8 to generate bounding box prompts for the Segment Anything Model (SAM) and High-Quality SAM (HQ-SAM), the authors achieve robust multimodal segmentation across X-ray, CT, and Ultrasound modalities with minimal training data. The hybrid YOLOv8+SAM approach demonstrates superior performance over standalone models, maintaining high Dice scores even with imprecise region-of-interest (ROI) detection.

Introduction

Identifying precise boundaries of organs or lesions is a cornerstone of medical imaging. While traditional methods rely on manual contouring or specialized U-Net architectures, these are often labor-intensive or limited by the trade-off between local detail and global context. The emergence of the Segment Anything Model (SAM) has provided a zero-shot solution for high-quality segmentation, yet its reliance on manual prompts (points or boxes) limits its clinical automation. This study addresses this limitation by proposing a fully automated pipeline that leverages YOLOv8 as a "prompt generator" for SAM, requiring only a small fraction of the labeled data typically needed for medical segmentation tasks.

Methodology

The proposed methodology follows a three-stage process designed for efficiency and multimodal versatility:

  1. Data Preprocessing: Standardizing images across X-ray, CT, and Ultrasound modalities to ensure consistent input for the detection model.
  2. ROI Detection via YOLOv8: Instead of training a complex segmentation model, a lightweight YOLOv8 model is trained to detect approximate bounding boxes around the target anatomy. The authors found that training on as few as 100 image-mask pairs was sufficient to produce effective prompts.
  3. Foundation Model Integration: The bounding boxes generated by YOLOv8 serve as visual prompts for SAM and HQ-SAM. These models then perform the final delineation of object boundaries.
Proposed automated segmentation pipeline using YOLOv8 as a prompt generator for SAM and HQ-SAM.

A key finding of this approach is the robustness to "prompt jitter." SAM’s performance remains stable even if the YOLOv8 bounding box is slightly inaccurate (e.g., off by 5–10 pixels), allowing for the use of a detector trained on very small datasets.

Experimental Setup

The pipeline was evaluated across three distinct datasets to demonstrate its multimodal capabilities:

  • Lung X-ray: 705 2D images from open-source Kaggle datasets.
  • Lung CT Scan: 267 2D images from open-source Kaggle datasets.
  • Short-Axis Aorta Ultrasound: 200 images from Chang Gung Memorial Hospital.

Results and Analysis

The hybrid YOLOv8 + SAM model consistently outperformed other configurations across all modalities.

  • X-ray: Achieved a Mean Dice Score of 0.9012. Standalone YOLOv8 segmentation failed significantly here, managing only a 0.1938 Dice score.
  • Lung CT: SAM reached a Mean Dice Score of 0.8799, notably outperforming the "High Quality" HQ-SAM variant (0.8554).
  • Ultrasound: Despite being the most challenging modality due to noise, SAM maintained a lead with a Mean Dice Score of 0.769.
Qualitative comparison of segmentation masks across three medical imaging modalities.

The visual results highlight a critical insight: YOLOv8’s native segmentation capabilities often produce crude, boxy masks. However, when used as a prompter, it unlocks SAM’s ability to capture fine-grained anatomical details. Interestingly, HQ-SAM did not consistently provide better results than the standard SAM in these medical contexts, suggesting that the standard model is sufficiently robust for general ROI segmentation.

Conclusion and Future Work

This research demonstrates that combining lightweight detectors with large-scale foundation models can drastically reduce the data requirements for high-quality medical segmentation. The YOLOv8+SAM pipeline offers a scalable path toward automated clinical tools.

Future research will focus on extending this 2D logic to 3D volumes (MRI/CT stacks), incorporating more diverse clinical datasets to handle class imbalances, and optimizing the computational overhead of large foundation models for real-time clinical applications.

Summary updated

BibTeX
@inproceedings{pandey2023comprehensive,
  author = {Sumit Pandey and Kuan-Fu Chen and Erik B. Dam},
  title = {{Comprehensive Multimodal Segmentation in Medical Imaging: Combining YOLOv8 with SAM and HQ-SAM Models}},
  booktitle = {Workshops @ ICCV},
  year = {2023},
  doi = {10.1109/iccvw60793.2023.00273},
}
Figure