TL;DR
This paper introduces MedVidQA 2022, a shared task designed to advance medical video question-answering by focusing on video classification and temporal answer localization. The task addresses the critical need for systems that can understand complex medical videos to provide accurate visual answers to health-related queries, a capability currently underdeveloped in AI.
What Problem Does This Paper Address?
The increasing availability of online medical videos presents a challenge: how can we effectively extract and present crucial information to users seeking answers to health-related questions? Existing video question-answering (VQA) systems often focus on open-domain videos like movies and aim to provide natural language answers. However, for medical queries, a textual answer alone can be insufficient. Users often need to see a specific demonstration or procedure within a video. This requires not only identifying the correct video but also pinpointing the exact temporal segment where the visual answer is presented. MedVidQA 2022 tackles this by proposing two tasks: classifying medical videos and localizing the precise visual answer within them.
Methods
The MedVidQA 2022 shared task comprised two main challenges:
- Medical Video Classification (MVC): This task involved categorizing videos into three classes: "Medical Instructional" (demonstrating a procedure for non-professionals), "Medical Non-instructional" (discussing medical topics without visuals), and "Non-medical." Participants developed systems to perform this classification.
- Medical Visual Answer Localization (MVAL): This task required systems to identify the specific temporal segment (start and end timestamps) within a medical video that visually answers a given medical question. This is akin to "visual grounding" but specifically tailored for medical contexts.
For baselines, the paper describes:
- Monomodal (Language) Baseline: Utilized English subtitles from videos, processed by a BERT-based model for classification.
- Monomodal (Vision) Baseline: Extracted features from video frames using a ViT model, followed by an LSTM for classification.
- Multimodal Baseline: Combined both subtitle and frame features for classification.
- VSL-BASE (MVAL): A multimodal fusion approach using Transformers to encode questions and video frames, then fusing them with attention to predict answer timestamps.
- VSL-QGH (MVAL): An extension of VSL-BASE that treats the target temporal segment as foreground.
Datasets
The paper utilizes datasets derived from existing large-scale video collections:
- MedVidCL: Used for training and validation of the MVC task. This dataset was constructed using a human-assisted two-stage approach, leveraging models trained on HowTo100M and YouTube8M datasets.
- MedVidQA: Created from top YouTube search results for WikiHow health-related queries. It contains medical instructional videos with expert-curated instructional questions and corresponding visual answer timestamps. The training and validation sets were derived from this, with a separate test set created similarly.
Key Insights and Ideas
- Subtitles are King for Classification: For the Medical Video Classification task, video subtitles proved to be the most dominant feature. Systems relying solely on subtitles, especially those using large pre-trained language models capable of handling long sequences, achieved high performance.
- Coarse-Grained Classification is Easier: The MVC task, focusing on broad categories, was found to be relatively easier than traditional fine-grained action recognition tasks.
- MVAL Benefits from Subtitles: In the Medical Visual Answer Localization task, video subtitles were more effective than raw video features for locating answers. This suggests that the spoken or transcribed explanations within videos are highly informative for pinpointing visual answers.
- MVAL Resembles Machine Reading Comprehension: The MVAL task's reliance on subtitles and questions to find relevant spans closely mirrors the challenges in machine reading comprehension (MRC) tasks.
Key Results
- MVC Task: The winning team, VPAI_LAB, achieved the highest Med-Inst F1-score of 98.74. The top-ranked participants significantly outperformed the baseline systems. Notably, the monomodal language baseline achieved the highest Med-Inst F1-score among all baseline approaches.
- MVAL Task: Team PAHT achieved the highest IoU=0.7 score of 73.20. The top-3 participating systems in MVAL also leveraged subtitles and pre-trained language models, similar to MRC approaches. Team LINGJING was the only team to propose a multimodal approach that achieved a competitive rank (4th).
Limitations and Future Work
- Multimodal Fusion for MVAL: While subtitles were dominant, the paper suggests that fusing language and vision features without losing information could further enhance performance, particularly for MVAL. The success of LINGJING's multimodal approach supports this.
- Dataset Scale and Diversity: While MedVidQA is a valuable contribution, future work could involve expanding the dataset with more diverse medical topics and video types.
- Complexity of Medical Queries: The current task focuses on specific types of medical questions. Future work could explore more complex, multi-step reasoning or diagnostic questions.
- Real-World Deployment: The paper highlights the potential for these systems to aid in clinical decision-making and patient education, suggesting that further research into robust and interpretable models is crucial for real-world deployment.
Summary updated