ScribbleVC: Scribble-supervised Medical Image Segmentation with Vision-Class Embedding

Zihan Li, Yuan Zheng, Xiangde Luo, Dandan Shan, Qingqi Hong

Introduction

Accurate medical image segmentation is a cornerstone of modern computer-aided diagnosis, treatment planning, and disease monitoring. However, the development of robust deep learning models for this task is frequently hindered by the "annotation bottleneck." Traditional fully-supervised methods require dense, pixel-perfect masks—a labor-intensive process where medical experts must meticulously trace anatomical boundaries. Scribble-based supervision offers a pragmatic alternative, requiring only sparse lines to indicate object categories. While significantly reducing annotation time, these sparse labels often lead to poor boundary definition and shape inconsistency.

In this paper, we present ScribbleVC, a novel framework designed to bridge the performance gap between weak scribble supervision and full supervision. By integrating category-level information directly into the visual feature extraction process through a "Vision-Class Embedding" mechanism, ScribbleVC achieves state-of-the-art results across multiple medical imaging benchmarks.

Methodology

ScribbleVC addresses the limitations of sparse labels through a multi-faceted architecture that combines global context, local texture, and semantic category information.

1. Hybrid CNN-Transformer Encoder

To capture both fine-grained local details and broad anatomical structures, ScribbleVC employs a hybrid encoder. Convolutional Neural Networks (CNNs) excel at extracting local textures, while Transformers utilize self-attention mechanisms to model long-range dependencies and global shapes. By combining these, the model maintains a comprehensive understanding of the image.

2. Vision-Class (VC) Embedding

The core innovation of ScribbleVC is the Vision-Class (VC) Embedding. In scribble-supervised learning, the category of the scribble (e.g., "Left Ventricle") provides a strong semantic hint. ScribbleVC learns a "class embedding" for each category and fuses these embeddings with the visual features. This process effectively provides the network with a "prior" or a "hint" about what features to prioritize in specific regions, helping it distinguish between similar-looking tissues.

3. Multimodal Information Enhancement

The framework treats class labels as a distinct modality, similar to how text is treated in vision-language models. This multimodal approach refines visual features and facilitates the generation of high-quality pseudo-labels for pixels that were not originally annotated. This iterative refinement allows the model to "fill in the gaps" left by the sparse scribbles.

4. Dual-Branch Supervision and Dynamic Mixing

The architecture utilizes two separate decoders—one for CNN-derived features and one for Transformer-derived features. A consistency-based supervision strategy ensures that both branches agree on the final segmentation. Furthermore, a "dynamic mixing" strategy is employed for pseudo-labeling: training signals are only generated for unlabeled pixels when both branches exhibit high confidence (above a threshold of 0.5), reducing the impact of noise.

Experimental Results

The ScribbleVC framework was evaluated on three major benchmarks: the ACDC (cardiac MRI), MSCMRseg (LGE MRI), and NCI-ISBI (prostate MRI) datasets.

Performance Benchmarks

ScribbleVC established new state-of-the-art (SOTA) performance for scribble-supervised learning. On the ACDC dataset, it achieved a Dice score of 88.4%, a significant improvement over the previous SOTA, CycleMix (84.8%).

Comparison with Full Supervision

One of the most striking findings was that ScribbleVC outperformed several established fully-supervised models, such as UNet++, on the MSCMRseg dataset. This suggests that the integration of class embeddings and multimodal refinement can compensate for the lack of dense pixel-level labels.

Data Efficiency

Sensitivity analyses demonstrated the model's robustness in low-data scenarios. ScribbleVC achieved a 75.1% accuracy score with only 20 scribble-annotated training samples, highlighting its potential for clinical applications where data is scarce.

Discussion and Limitations

While ScribbleVC represents a significant advancement, certain limitations remain. The model's performance during inference is partially dependent on "historical category feature vectors" derived during training, which may introduce bias if the training set is not representative. Additionally, the reliance on a fixed confidence threshold for pseudo-labeling suggests that future iterations could benefit from automated, adaptive thresholding. Finally, while the model handles 3D datasets, it primarily processes them as 2D slices; extending the multimodal logic into native 3D space could further enhance spatial consistency.

Conclusion

ScribbleVC demonstrates that by treating category labels as an active component of the feature extraction process rather than just a passive target, we can significantly enhance the performance of weakly-supervised medical image segmentation. The framework provides a viable path toward high-accuracy diagnostic tools that do not require the prohibitive cost of dense manual annotation.

Summary updated

BibTeX
@inproceedings{li2023scribblevc,
  author = {Zihan Li and Yuan Zheng and Xiangde Luo and Dandan Shan and Qingqi Hong},
  title = {{ScribbleVC: Scribble-supervised Medical Image Segmentation with Vision-Class Embedding}},
  booktitle = {ACM Multimedia},
  year = {2023},
  doi = {10.1145/3581783.3612056},
}
Figure