Introduction
In clinical diagnostic workflows, particularly during endoscopic procedures, medical professionals require two primary forms of assistance from artificial intelligence: the identification of abnormalities (detection) and the precise delineation of their boundaries (segmentation). While specialized single-task networks have achieved high performance in these areas, running multiple models sequentially is computationally intensive and often fails to meet the requirements for real-time clinical application.
YOLO-MED is a novel, end-to-end multi-task network designed to bridge this gap. By performing object detection and semantic segmentation simultaneously within a unified architecture, it achieves state-of-the-art accuracy on polyp and gastric lesion datasets while maintaining the inference speeds necessary for live medical procedures.
The Challenge: Accuracy vs. Speed
Existing multi-task networks typically utilize a shared "backbone" for feature extraction to save on computation. However, they frequently encounter two significant hurdles:
- The Accuracy-Speed Trade-off: Maintaining high precision while ensuring the model is fast enough for real-time use (e.g., during a colonoscopy) is a persistent challenge.
- Feature Neglect: Biomedical objects, such as lesions or polyps, often exhibit subtle visual differences from surrounding healthy tissue. Standard models often fail to integrate "cross-scale" information—the ability to use global context to inform the identification of minute, localized details.
Architectural Innovations
YOLO-MED moves beyond simple shared encoders by introducing a more interactive and decoupled architecture. Its design is centered on three core components:
1. Efficient Backbone and Neck
The model utilizes a CSPDarknet53 backbone for efficient feature extraction, paired with a Feature Pyramid Network (FPN). This combination allows the network to fuse features across different semantic levels, ensuring that both high-level conceptual data and low-level structural data are preserved.
2. Decoupled Detection Head
Standard YOLO architectures often use a single "head" to handle both classification (identifying the object) and regression (locating the bounding box). YOLO-MED separates these tasks into distinct branches. This decoupling reduces "task conflict," a common phenomenon in multi-task learning where the requirements of one task degrade the performance of another.
3. Cross-Scale Task-Interaction (CSTI) Module
The CSTI module is the primary innovation of YOLO-MED. It treats feature maps from both the detection and segmentation decoders as "tokens" and processes them through a Transformer Layer. This allows the network to learn complex, non-linear relationships between the two tasks. For instance, high-resolution detection features can be used to refine the boundaries of a low-resolution segmentation mask, effectively using the strengths of one task to compensate for the weaknesses of the other.
Experimental Results
The researchers evaluated YOLO-MED on two challenging benchmarks: the public Kvasir-seg dataset (1,000 gastrointestinal polyp images) and a Private Gastric Dataset (3,757 images of gastric neoplastic lesions) developed in collaboration with Shanghai General Hospital.
Quantitative Performance
YOLO-MED demonstrated superior performance compared to both specialized single-task models and established multi-task frameworks like UOLO and MULAN:
- Accuracy (Kvasir-seg): The model achieved a 97.32% Pixel Accuracy (PA) for segmentation and a 94.72% mAP50 for detection. This represents a significant improvement over the multi-task baseline, which recorded 90.88% PA and 89.73% mAP50.
- Inference Speed: Despite the architectural complexity introduced by the CSTI module, YOLO-MED maintains a frame rate of 31 FPS on an NVIDIA 2080Ti. While this is a slight decrease from the 36 FPS baseline, it remains well above the threshold required for real-time clinical utility.
Qualitative Advantages
Visual analysis indicates that YOLO-MED is particularly effective at identifying multiple small objects and accurately defining the margins of large, irregular lesions—areas where previous models often struggled.
Conclusion and Future Work
YOLO-MED proves that Transformers can be used effectively as a communication bridge between detection and segmentation tasks rather than just for global context. By decoupling task heads and facilitating cross-scale interaction, the model achieves a high-performance balance suitable for the operating room.
Future research will focus on optimizing the CSTI module to further reduce the computational overhead and exploring the application of the YOLO-MED architecture to 3D modalities, such as CT or MRI scans, where cross-scale interaction may yield even more significant diagnostic benefits.
Summary updated