TL;DR
This study systematically debunks the common belief that plain Vision Transformers (ViTs) are inherently unsuitable for segmenting thin structures like retinal vessels. By comparing pre-trained foundation models like MedSAM against specialized CNNs, the authors demonstrate that with sufficient medical pre-training, ViTs can match or exceed CNN performance, though often at a significantly higher computational cost.
What Problem Does This Paper Address?
Retinal vessel segmentation is a cornerstone of ophthalmic diagnostics, used to track diseases like diabetic retinopathy and glaucoma. However, "thin vessels"—those only a few pixels wide—remain a major hurdle. They have low contrast and contribute little to standard overlap metrics like the Dice score.
Conventional wisdom in the field suggests that plain Vision Transformers (ViTs) should struggle with these fine structures. The reasoning is twofold: first, ViTs divide images into coarse patches (e.g., 16x16 pixels), which might "average out" thin vessels; second, they lack the "locality bias" of Convolutional Neural Networks (CNNs), making them supposedly less precise for dense, pixel-level tasks. This paper sets out to quantify whether these theoretical weaknesses actually translate to poor performance in practice.
Methods
The researchers conducted a rigorous head-to-head comparison between several architectures:
- MedSAM: A "plain" ViT-B encoder pre-trained on over a million medical image-mask pairs.
- LiteMedSAM: A hybrid model combining convolutional stages with windowed attention, distilled from MedSAM.
- Three CNNs: Ranging from the massive ConvNeXt-L (211M parameters) to the tiny LW-Net (just 0.07M parameters).
Crucially, the team moved beyond the standard Dice score and focused on centerline recall ($R_{thin}$) and precision ($P_{thin}$). By skeletonizing the segmentation masks, they could specifically measure how well each model captured the actual topology of the thin vessel network (vessels with a radius < 1.5 pixels).
Datasets
The models were evaluated across four diverse datasets to ensure the findings weren't domain-specific:
- DRIVE: The classic retinal benchmark with very few training images (16).
- HRF: High-resolution retinal images (3504 × 2336 pixels).
- FIVES: A larger retinal dataset (600 training images) featuring various pathologies.
- VessMap: A microscopy dataset of mouse brain vasculature, used as a "second-domain" check.
Key Insights and Ideas
- Pre-training is the "Secret Sauce": The most significant finding is that large-scale medical pre-training allows plain ViTs to overcome their architectural limitations. Without it, MedSAM’s performance on thin vessels was significantly worse than the CNNs.
- The Patch Size Myth: Despite using 16x16 pixel patches, MedSAM successfully detected vessels much thinner than a single patch. This suggests that the transformer's global attention can effectively reconstruct fine details if the weights are properly initialized.
- Hybrid Models for High Recall: LiteMedSAM (the hybrid) consistently achieved the highest recall for thin vessels. However, as shown in the precision-recall plots, it often achieved this by over-segmenting, leading to lower precision than the best CNNs.
- Sledgehammer vs. Scalpel: While MedSAM is powerful, it is a computational "sledgehammer." The tiny LW-Net achieved comparable results on several datasets while being orders of magnitude smaller and more efficient.
Key Results
- Performance Parity: Fine-tuned MedSAM's centerline recall for thin vessels was not significantly different from the best CNN across all four datasets, with differences ranging from only -0.6 to +2.5 percentage points.
- The Cost of Attention: MedSAM is computationally expensive. It required 2 to 18 times more GFLOPs per image than the medium-sized ConvUNeXt and over 30 times more than the tiny LW-Net.
- Inference Protocol Matters: For MedSAM, upscaling smaller images to its native 1024x1024 training resolution was critical. Using a simple "padded" protocol instead of "upscaled" led to a massive drop in recall (e.g., -41.3 points on VessMap).
Limitations and Future Work
The authors acknowledge that thin vessels are notoriously difficult even for humans to label. On the DRIVE dataset, the agreement between two human annotators on thin vessels was only ~58.6%, which is remarkably close to the models' performance. This suggests that the field may be approaching a "label noise ceiling" where further architectural improvements are hard to measure.
Additionally, while the Transformers benefited from massive pre-training, the CNNs were trained from scratch (reflecting common practice for compact medical models). Future work could investigate if pre-trained CNNs might reclaim the lead in efficiency-adjusted performance.
Summary updated