Introduction
Accurate medical image segmentation is a cornerstone of modern computer-aided diagnosis, treatment planning, and disease monitoring. However, the development of robust deep learning models for this task is frequently hindered by the "annotation bottleneck." Traditional fully-supervised methods require dense, pixel-perfect masks—a labor-intensive process where medical experts must meticulously trace anatomical boundaries. Scribble-based supervision offers a pragmatic alternative, requiring only sparse lines to indicate object categories. While significantly reducing annotation time, these sparse labels often lead to poor boundary definition and shape inconsistency.
In this paper, we present ScribbleVC, a novel framework designed to bridge the performance gap between weak scribble supervision and full supervision. By integrating category-level information directly into the visual feature extraction process through a "Vision-Class Embedding" mechanism, ScribbleVC achieves state-of-the-art results across multiple medical imaging benchmarks.
Methodology
ScribbleVC addresses the limitations of sparse labels through a multi-faceted architecture that combines global context, local texture, and semantic category information.
1. Hybrid CNN-Transformer Encoder
To capture both fine-grained local details and broad anatomical structures, ScribbleVC employs a hybrid encoder. Convolutional Neural Networks (CNNs) excel at extracting local textures, while Transformers utilize self-attention mechanisms to model long-range dependencies and global shapes. By combining these, the model maintains a comprehensive understanding of the image.
2. Vision-Class (VC) Embedding
The core innovation of ScribbleVC is the Vision-Class (VC) Embedding. In scribble-supervised learning, the category of the scribble (e.g., "Left Ventricle") provides a strong semantic hint. ScribbleVC learns a "class embedding" for each category and fuses these embeddings with the visual features. This process effectively provides the network with a "prior" or a "hint" about what features to prioritize in specific regions, helping it distinguish between similar-looking tissues.
3. Multimodal Information Enhancement
The framework treats class labels as a distinct modality, similar to how text is treated in vision-language models. This multimodal approach refines visual features and facilitates the generation of high-quality pseudo-labels for pixels that were not originally annotated. This iterative refinement allows the model to "fill in the gaps" left by the sparse scribbles.
4. Dual-Branch Supervision and Dynamic Mixing
The architecture utilizes two separate decoders—one for CNN-derived features and one for Transformer-derived features. A consistency-based supervision strategy ensures that both branches agree on the final segmentation. Furthermore, a "dynamic mixing" strategy is employed for pseudo-labeling: training signals are only generated for unlabeled pixels when both branches exhibit high confidence (above a threshold of 0.5), reducing the impact of noise.
Experimental Results
The ScribbleVC framework was evaluated on three major benchmarks: the ACDC (cardiac MRI), MSCMRseg (LGE MRI), and NCI-ISBI (prostate MRI) datasets.
Performance Benchmarks
ScribbleVC established new state-of-the-art (SOTA) performance for scribble-supervised learning. On the ACDC dataset, it achieved a Dice score of 88.4%, a significant improvement over the previous SOTA, CycleMix (84.8%).
Comparison with Full Supervision
One of the most striking findings was that ScribbleVC outperformed several established fully-supervised models, such as UNet++, on the MSCMRseg dataset. This suggests that the integration of class embeddings and multimodal refinement can compensate for the lack of dense pixel-level labels.
Data Efficiency
Sensitivity analyses demonstrated the model's robustness in low-data scenarios. ScribbleVC achieved a 75.1% accuracy score with only 20 scribble-annotated training samples, highlighting its potential for clinical applications where data is scarce.
Discussion and Limitations
While ScribbleVC represents a significant advancement, certain limitations remain. The model's performance during inference is partially dependent on "historical category feature vectors" derived during training, which may introduce bias if the training set is not representative. Additionally, the reliance on a fixed confidence threshold for pseudo-labeling suggests that future iterations could benefit from automated, adaptive thresholding. Finally, while the model handles 3D datasets, it primarily processes them as 2D slices; extending the multimodal logic into native 3D space could further enhance spatial consistency.
Conclusion
ScribbleVC demonstrates that by treating category labels as an active component of the feature extraction process rather than just a passive target, we can significantly enhance the performance of weakly-supervised medical image segmentation. The framework provides a viable path toward high-accuracy diagnostic tools that do not require the prohibitive cost of dense manual annotation.
Summary updated