CCT-Net: Category-Invariant Cross-Domain Transfer for Medical Single-to-Multiple Disease Diagnosis

Yi Zhou, Lei Huang, Tao Zhou, Ling ShaoICCV 2021 · OpenReview

Abstract

CCT-Net addresses the "annotation bottleneck" in medical AI by transferring knowledge from a well-studied disease with pixel-level labels (source) to multiple related diseases that only have image-level labels (target). By learning a category-invariant way to refine coarse disease localizations into fine-grained masks, the model significantly improves both the localization and classification of new diseases without needing additional expensive manual segmentations.

Introduction

In medical imaging, expert pixel-level annotation is time-consuming and expensive. Consequently, while "rich" datasets exist for common conditions like Diabetic Retinopathy, clinical reality requires diagnosing multiple diseases simultaneously, many of which may only have simple image-level tags. Existing transfer learning methods often require source and target diseases to be identical or necessitate extensive fine-tuning. CCT-Net tackles the single-to-multiple disease diagnosis challenge: leveraging the "spatial intelligence" learned from one well-annotated disease to identify and localize several related diseases that lack fine-grained maps.

Methodology

The CCT-Net framework consists of three primary components designed to bridge the gap between source and target domains while refining disease localization.

  1. Domain-Specific Encoder (DSE): To account for style shifts between different scanners or populations, the model utilizes domain-specific Batch Normalization (BN). This allows the network to learn shared medical features while maintaining domain-specific statistics.
  2. Confidence Weighted Pooling (CWP): Unlike standard Global Average Pooling, CWP is a trainable module that assigns weights to image regions based on disease probability. This results in sharper and more accurate coarse heatmaps (Class Activation Maps).
  3. Category-Invariant Feature Refinement (CIFR): This is the core innovation. CIFR blocks learn a general refinement task: producing a pixel-level mask given a feature map and a coarse heatmap. Because the process is conditioned on the heatmap rather than a specific class, it is category-invariant and can be applied to entirely different diseases in the target domain.
The CCT-Net architecture, highlighting the Domain-Specific Encoder (DSE), Confidence Weighted Pooling (CWP), and the Category-Invariant Feature Refinement (CIFR) blocks.

Refined features from this process are subsequently processed by a second encoder (DSE-2) to enhance final image-level classification accuracy.

Experimental Results

The model was evaluated on two distinct medical imaging tasks: Ocular Disease diagnosis (Fundus images) and Brain Tumor classification (MRI).

  • Ocular Task: Using FGADR (pixel-level Diabetic Retinopathy) as the source and ODIR-5K (image-level for 8 diseases) as the target, CCT-Net achieved a Kappa score of 0.7508, outperforming standard baselines like DANN (0.6934) and MCD (0.7296).
  • Brain Task: Using BraTS 2019 (pixel-level glioma) as the source and a multi-class BrainTumor dataset as the target, the model reached an average accuracy of over 95% across meningioma, pituitary tumors, and glioma.
Qualitative comparison of input images, coarse heatmaps, and refined masks against ground truth for both source and target domains.

Notably, the model generated accurate segmentations for diseases it never saw pixel-level masks for during training (e.g., meningiomas), demonstrating the effectiveness of the category-invariant refinement.

Discussion and Conclusion

The primary contribution of CCT-Net is treating segmentation refinement as a category-agnostic skill. By conditioning refinement on coarse heatmaps, the network learns a general geometric logic that transfers across diseases. This approach demonstrates that improved localization directly enhances classification robustness. Future work may explore cross-modality transfer and more advanced weakly-supervised localization methods to further improve the initial "seed" heatmaps.

Summary updated

BibTeX
@inproceedings{zhou2021cct,
  author = {Yi Zhou and Lei Huang and Tao Zhou and Ling Shao},
  title = {{CCT-Net: Category-Invariant Cross-Domain Transfer for Medical Single-to-Multiple Disease Diagnosis}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  url = {https://openreview.net/forum?id=H0gzopbxnq},
}
Figure