Text-Guided Foundation Model Adaptation for Long-Tailed Medical Image Classification

Sirui Li, Li Lin, Yijin Huang, Pujin Cheng, Xiaoying Tang

TL;DR

This paper introduces TFA-LT, a novel method to improve medical image classification on imbalanced (long-tailed) datasets by adapting pre-trained vision-language foundation models. TFA-LT uses a two-stage training strategy with lightweight adapters and an ensembler, achieving significant accuracy gains with drastically reduced computational resources.

What Problem Does This Paper Address?

Medical datasets are often "long-tailed," meaning common diseases have abundant data, while rare diseases have very few examples. This imbalance severely hinders the performance of deep learning models, leading to biased predictions that can miss critical rare conditions. While large multimodal foundation models (like CLIP) show promise for data scarcity, their general pre-training on natural images limits their effectiveness in specialized medical domains. Existing methods for long-tailed learning often struggle with computational cost or performance trade-offs between common and rare classes. This paper aims to bridge this gap by effectively adapting these powerful foundation models for the specific challenges of long-tailed medical image classification.

Methods

The core of TFA-LT is a two-stage training strategy:

  • Stage I: Residual Connection Adapter: The authors freeze the pre-trained foundation model's encoders. They introduce a "residual connection adapter" which uses a linear adapter and a learnable hyperparameter to adjust visual features. This adapter aims to better align visual representations with richer textual descriptions of the medical conditions, moving beyond simple one-hot labels. To further balance the data, they employ two sampling strategies: Weighted Random Sampling (WRS) to boost rare classes and Random Under Sampling (RUS) to reduce common classes. Textual prompts are also enhanced to be more descriptive (e.g., "The category of this [data type] image is [class name].").
  • Stage II: Representation Ensemble Training: After Stage I, the adapters are frozen. In Stage II, the original, imbalanced dataset is used. A single linear ensembler is trained to combine the representations from the two models trained in Stage I (one using WRS, the other RUS). This ensembler can operate at either the "logit level" (combining the final prediction scores) or the "feature level" (combining intermediate feature representations). Focal loss is used during this stage to further fine-tune for the imbalanced data.

Datasets

The proposed method is evaluated on two publicly available, long-tailed medical image datasets:

  • ISIC2018: A large collection of dermoscopy images for skin lesion analysis.
  • APTOS2019: A fundus dataset for diabetic retinopathy grading.

The datasets are split into an 8:2 training-test ratio, with the test set maintained in a balanced distribution. Both datasets are further divided into "many," "medium," and "few" subsets to analyze performance on head, middle, and tail classes, respectively.

Key Insights and Ideas

  • Textual Prompts as Richer Labels: Moving beyond simple class names to descriptive text prompts for foundation models can significantly improve their understanding and performance in specialized domains like medicine.
  • Two-Stage Adaptation for Balance: A staged approach, first focusing on aligning and balancing representations with adapters and then ensembling these balanced views, proves more effective than a single-stage fine-tuning.
  • Lightweight Adaptation is Key: The success of TFA-LT hinges on using minimal, lightweight components (two linear adapters and one ensembler) that leverage the power of the pre-trained foundation model without requiring extensive retraining.
  • Ensembling for Robustness: Combining representations from models trained with different sampling strategies (WRS and RUS) effectively mitigates the biases introduced by each individual strategy, leading to more robust predictions.

Key Results

  • Significant Accuracy Improvement: TFA-LT achieves an accuracy improvement of up to 27.1% on the APTOS2019 dataset compared to the current best-performing algorithm. It also shows competitive results on ISIC2018.
  • Drastic Reduction in Computational Cost: The method requires only 6.1% of the GPU memory usage of the leading algorithm (BCL) during training. Specifically, Stage I uses 2.56GB and Stage II uses 2.75GB of GPU memory.
  • State-of-the-Art Performance: TFA-LT surpasses other state-of-the-art long-tailed classification algorithms on APTOS2019 and achieves comparable results on ISIC2018, demonstrating its effectiveness.

Limitations and Future Work

  • Generalizability: While TFA-LT shows strong performance on the tested medical datasets, its generalizability to other medical imaging modalities or even other long-tailed domains might require further investigation.
  • Foundation Model Dependence: The performance is inherently tied to the quality and pre-training of the chosen foundation model (CLIP in this case).
  • Future Work: The authors suggest that this approach highlights the potential of foundation model adaptation for long-tailed challenges, implying that further exploration of different adaptation strategies and foundation models could yield even better results. The simplicity and efficiency of TFA-LT make it a promising direction for future research in medical AI.

Summary updated

Abstract

In medical contexts, the imbalanced data distribution in long-tailed datasets, due to scarce labels for rare diseases, greatly impairs the diagnostic accuracy of deep learning models. Recent multimodal text-image supervised foundation models offer new solutions to data scarcity through effective representation learning. However, their limited medical-specific pretraining hinders their performance in medical image classification relative to natural images. To address this issue, we propose a novel Text-guided Foundation model Adaptation for Long-Tailed medical image classification (TFA-LT). We adopt a two-stage training strategy, integrating representations from the foundation model using just two linear adapters and a single ensembler for balanced outcomes. Experimental results on two long-tailed medical image datasets validate the simplicity, lightweight and efficiency of our approach: requiring only 6.1% GPU memory usage of the current best-performing algorithm, our method achieves an accuracy improvement of up to 27.1%, highlighting the substantial |potential of foundation model adaptation in this area.
BibTeX
@inproceedings{li2024text,
  author = {Sirui Li and Li Lin and Yijin Huang and Pujin Cheng and Xiaoying Tang},
  title = {{Text-Guided Foundation Model Adaptation for Long-Tailed Medical Image Classification}},
  booktitle = {ISBI},
  year = {2024},
  url = {https://openreview.net/forum?id=XIlRS6GVqW},
}
Figure