MediFact at MEDIQA-M3G 2024: Medical Question Answering in Dermatology with Multimodal Learning

Nadia SaeedNAACL 2024

TL;DR

This paper introduces MediFact-M3G, a novel multimodal learning framework for generating informative answers to dermatology-related medical questions. It tackles the challenge of limited labeled data by employing weakly supervised learning for image representation and a multi-model approach for response generation, ultimately aiming to improve clinical decision support systems.

What Problem Does This Paper Address?

Dermatological telemedicine offers a promising avenue for remote diagnosis and treatment, but it faces significant hurdles. Traditional approaches often struggle with capturing subtle visual details of skin conditions and the inability to physically examine lesions, leading to miscommunication and potentially ineffective treatment plans. Existing consumer health question-answering systems primarily rely on textual data, neglecting the crucial visual information present in medical images. While some research has explored combining clinical text and images for specific dermatology tasks, there's a gap in addressing open-ended, multilingual medical question answering, especially when dealing with limited labeled data. This paper aims to bridge this gap by developing a system that can effectively answer complex, open-ended questions using both visual and textual information across multiple languages.

Methods

The MediFact-M3G framework employs a multi-stage approach:

  • Data Preprocessing and Response Weighting: The initial step involves cleaning and standardizing the raw data. A crucial aspect here is a weighting function that assigns scores to responses based on factors like author expertise and completeness. This guides the model to prioritize learning from more reliable and informative answers.
  • Weakly Supervised Learning for Image Representation: To overcome the limitations of scarce labeled data, the system uses a pre-trained VGG16 Convolutional Neural Network (CNN) to extract high-level features from dermatological images. These features are then fed into a Support Vector Machine (SVM) classifier, which learns the relationship between image features and corresponding textual descriptions. This allows the model to generalize learned relationships to unlabeled images and enrich image representations with semantic meaning.
  • Multi-Model Response Generation with Feature Fusion: This stage combines pre-trained question-answering (QA) models with the image representations learned in the previous step. It utilizes both an extractive QA model (to retrieve relevant passages) and an abstractive QA model (to generate comprehensive answers). The system creates a comprehensive feature vector by fusing the user's query, relevant textual content, and the image representation. This approach aims to ground answers in factual information while allowing for flexible generation.
  • Response Selection with Contrastive Learning: To select the most informative response, especially in multilingual settings, the framework leverages CLIP (Contrastive Language-Image Pre-Training). CLIP, with its Vision Transformer (ViT), extracts image features and uses a text encoder to calculate the cosine similarity between image embeddings and potential response embeddings. This allows for selecting the response with the closest semantic relationship to the image. For multilingual support, English responses are translated into Spanish and Chinese.

Datasets

The primary dataset used is MEDIQA-M3G 2024. This dataset is divided into training (842 instances), validation (56 instances), and test (100 instances) sets. Each set is available in Chinese, English, and Spanish versions. While non-English training sets are machine-translated, validation and test sets are human-translated for accuracy. Each instance includes a unique encounter ID, image IDs, query title and content, and author information.

Key Insights and Ideas

  • Weakly Supervised Learning for Image Representation is Crucial: The paper highlights that effective image representation can be learned even with limited labeled data by leveraging pre-trained models and SVMs, which is a significant finding for medical image analysis tasks where data is often scarce.
  • Multimodal Fusion Enhances Answer Quality: Combining visual and textual information through feature fusion leads to more accurate and informative responses, demonstrating the power of integrating different data modalities for complex medical queries.
  • Contrastive Learning for Robust Response Selection: Using CLIP for response selection proves effective in identifying the most semantically relevant answer, especially in multilingual contexts, by aligning image and text representations.
  • Translation as a Viable Strategy for Multilingual Support: The study shows that translating high-performing responses from one language (e.g., Chinese) to others can yield competitive results, offering a practical approach to multilingual medical QA.

Key Results

The MediFact-M3G framework achieved competitive results in the MEDIQA-M3G 2024 shared task, securing:

  • 7th rank in English language response generation.
  • 3rd rank in Chinese and Spanish language response generation.

These rankings were achieved out of a total of 75 participants. The paper also presents comparative results for different feature extraction models (VGG16, ResNet, SqueezNet), indicating that translated responses from the best-performing model (Chinese VGG16) showed improvements, and SqueezNet demonstrated exceptional proficiency in Chinese translations.

Limitations and Future Work

Limitations:

  • The paper acknowledges that while the translated version of MediFact-M3G showed slight improvements in BERT_Score, the individual setup performed better for Spanish language responses in terms of Deltableu score.
  • Potential translation inaccuracies are noted when converting English responses to Spanish and Chinese.
  • The study primarily focuses on English responses in the multi-model response generation stage due to limitations in non-English language models.

Future Work:

  • Explore the robustness and scalability of the approach with larger and more diverse datasets.
  • Investigate the integration of domain-specific ontologies and medical terminologies to enhance semantic understanding and accuracy.
  • Explore novel techniques for handling multi-turn dialogue scenarios to enable more natural and interactive conversations.
  • Collaborate with medical professionals to validate the clinical relevance and effectiveness of the system in real-world healthcare settings.

Summary updated

BibTeX
@inproceedings{saeed2024medifact,
  author = {Nadia Saeed},
  title = {{MediFact at MEDIQA-M3G 2024: Medical Question Answering in Dermatology with Multimodal Learning}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure