YOLO-Med : Multi-Task Interaction Network for Biomedical Images

Suizhi Huang, Shalayiding Sirejiding, Yuxiang Lu, Yue Ding, Leheng Liu, Hui Zhou, Hongtao Lu

Introduction

In clinical diagnostic workflows, particularly during endoscopic procedures, medical professionals require two primary forms of assistance from artificial intelligence: the identification of abnormalities (detection) and the precise delineation of their boundaries (segmentation). While specialized single-task networks have achieved high performance in these areas, running multiple models sequentially is computationally intensive and often fails to meet the requirements for real-time clinical application.

YOLO-MED is a novel, end-to-end multi-task network designed to bridge this gap. By performing object detection and semantic segmentation simultaneously within a unified architecture, it achieves state-of-the-art accuracy on polyp and gastric lesion datasets while maintaining the inference speeds necessary for live medical procedures.

The Challenge: Accuracy vs. Speed

Existing multi-task networks typically utilize a shared "backbone" for feature extraction to save on computation. However, they frequently encounter two significant hurdles:

  1. The Accuracy-Speed Trade-off: Maintaining high precision while ensuring the model is fast enough for real-time use (e.g., during a colonoscopy) is a persistent challenge.
  2. Feature Neglect: Biomedical objects, such as lesions or polyps, often exhibit subtle visual differences from surrounding healthy tissue. Standard models often fail to integrate "cross-scale" information—the ability to use global context to inform the identification of minute, localized details.

Architectural Innovations

YOLO-MED moves beyond simple shared encoders by introducing a more interactive and decoupled architecture. Its design is centered on three core components:

1. Efficient Backbone and Neck

The model utilizes a CSPDarknet53 backbone for efficient feature extraction, paired with a Feature Pyramid Network (FPN). This combination allows the network to fuse features across different semantic levels, ensuring that both high-level conceptual data and low-level structural data are preserved.

2. Decoupled Detection Head

Standard YOLO architectures often use a single "head" to handle both classification (identifying the object) and regression (locating the bounding box). YOLO-MED separates these tasks into distinct branches. This decoupling reduces "task conflict," a common phenomenon in multi-task learning where the requirements of one task degrade the performance of another.

3. Cross-Scale Task-Interaction (CSTI) Module

The CSTI module is the primary innovation of YOLO-MED. It treats feature maps from both the detection and segmentation decoders as "tokens" and processes them through a Transformer Layer. This allows the network to learn complex, non-linear relationships between the two tasks. For instance, high-resolution detection features can be used to refine the boundaries of a low-resolution segmentation mask, effectively using the strengths of one task to compensate for the weaknesses of the other.

Experimental Results

The researchers evaluated YOLO-MED on two challenging benchmarks: the public Kvasir-seg dataset (1,000 gastrointestinal polyp images) and a Private Gastric Dataset (3,757 images of gastric neoplastic lesions) developed in collaboration with Shanghai General Hospital.

Quantitative Performance

YOLO-MED demonstrated superior performance compared to both specialized single-task models and established multi-task frameworks like UOLO and MULAN:

  • Accuracy (Kvasir-seg): The model achieved a 97.32% Pixel Accuracy (PA) for segmentation and a 94.72% mAP50 for detection. This represents a significant improvement over the multi-task baseline, which recorded 90.88% PA and 89.73% mAP50.
  • Inference Speed: Despite the architectural complexity introduced by the CSTI module, YOLO-MED maintains a frame rate of 31 FPS on an NVIDIA 2080Ti. While this is a slight decrease from the 36 FPS baseline, it remains well above the threshold required for real-time clinical utility.

Qualitative Advantages

Visual analysis indicates that YOLO-MED is particularly effective at identifying multiple small objects and accurately defining the margins of large, irregular lesions—areas where previous models often struggled.

Conclusion and Future Work

YOLO-MED proves that Transformers can be used effectively as a communication bridge between detection and segmentation tasks rather than just for global context. By decoupling task heads and facilitating cross-scale interaction, the model achieves a high-performance balance suitable for the operating room.

Future research will focus on optimizing the CSTI module to further reduce the computational overhead and exploring the application of the YOLO-MED architecture to 3D modalities, such as CT or MRI scans, where cross-scale interaction may yield even more significant diagnostic benefits.

Summary updated

BibTeX
@inproceedings{huang2024yolo,
  author = {Suizhi Huang and Shalayiding Sirejiding and Yuxiang Lu and Yue Ding and Leheng Liu and Hui Zhou and Hongtao Lu},
  title = {{YOLO-Med : Multi-Task Interaction Network for Biomedical Images}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2024},
  doi = {10.1109/icassp48485.2024.10446165},
}
Figure