Introduction
YouTube has become a primary source of health information for the general public, yet the quality of its content remains highly variable. With only 12% of the US population possessing proficient health literacy, many users are vulnerable to misleading or low-quality medical advice found in user-generated videos. While expert-led clinical tutorials exist, they are often buried under a mountain of anecdotal "homemade" content.
Because the volume of uploads makes manual expert review impossible, researchers have developed an automated, multimodal deep learning framework to evaluate the medical knowledge level of these videos. This system aims to bridge the health literacy gap by algorithmically distinguishing between high-quality educational content and low-knowledge videos with high accuracy.
Methodology
The proposed framework utilizes an interdisciplinary approach that extracts knowledge from both the audio-visual and textual components of a video. The process is divided into three main stages: text-based extraction, image-based extraction, and final classification.
Text-based Medical Knowledge Assessment
To process the linguistic content, the system uses Natural Language Processing (NLP) techniques:
- Word Embeddings: Word2vec (Skip-gram) was trained on nearly 20,000 video descriptions to create 50-dimensional semantic vectors, capturing the context of medical terminology.
- BLSTM-RNN: A Bidirectional Long Short-Term Memory (BLSTM) model performs Named Entity Recognition (NER). By analyzing context both before and after a word, it identifies specific medical terms (e.g., "pathogenesis" or "insulin") more effectively than traditional dictionary-matching tools.
Image-based Medical Knowledge Extraction
To understand the visual context, the system employs computer vision:
- CNN Architecture: A pre-trained Inception Convolutional Neural Network (CNN) performs object detection on video frames sampled every two seconds.
- Feature Engineering: The model identifies the top five object categories per frame, which are then filtered for medical relevance (e.g., recognizing medical devices or clinical settings) to serve as features for classification.
Data Collection and Annotation
The study focused on Diabetes, initially scraping 19,873 unique videos. A subset of 600 videos with closed captions was selected for detailed analysis. Two graduate researchers annotated medical terms and labeled knowledge levels, with a medical doctor serving as the domain expert to consolidate annotations. The inter-rater reliability was high, reaching 0.87 for term annotation and 0.92 for knowledge classification. Additionally, over 200,000 individual frames were extracted for the image-based analysis.
Key Results and Insights
The framework demonstrated significant improvements over existing industry standards. The BLSTM method for medical term extraction achieved a 92.9% F-measure, vastly outperforming MetaMap (28.9%).
For the final task of categorizing videos into "High" versus "Low" medical knowledge, the integrated Logistic Regression classifier achieved an overall accuracy of 85%.
Key insights from the study include:
- Context is King: Deep learning models like BLSTM are far superior to simple lexicons for user-generated content, which often uses informal or non-standard language.
- Multimodal Synergy: Combining what is said (text) with what is shown (images) provides a more holistic understanding of a video's educational value.
Limitations and Future Work
While promising, the current model is a binary classifier (High/Low). Future iterations aim to provide more granular quality scores to match varying levels of patient health literacy. Additionally, the researchers plan to fine-tune the vision models on specialized medical datasets to improve the recognition of clinical equipment and expand the scope beyond diabetes to other chronic conditions. The ultimate goal is to integrate these "quality scores" into search and recommendation algorithms to prioritize reputable health information for all users.
Summary updated