YouTube Videos for Public Health Literacy? A Machine Learning Pipeline to Curate Covid-19 Videos

Yawen Guo, Xiao Liu, Anjana Susarla, Rema PadmanMedInfo 2023 · DOI

TL;DR

This paper tackles the challenge of identifying high-quality, understandable COVID-19 educational videos on YouTube. They developed a machine learning pipeline that leverages video features and expert annotations to automatically curate these resources, aiming to improve public health literacy during health crises.

What Problem Does This Paper Address?

The COVID-19 pandemic underscored the critical need for accessible and accurate public health information. While YouTube offers a vast repository of health-related content, its quality and understandability vary wildly, making it difficult for the public and healthcare professionals to find reliable educational materials. Traditional text-based health information can also be challenging for individuals with lower literacy levels. This paper addresses the gap in automatically identifying and curating medically relevant and understandable YouTube videos to combat misinformation and enhance public health literacy.

Methods

The authors developed a machine learning pipeline to curate COVID-19 videos. Their approach involved:

  • Keyword Generation: Identifying relevant search terms from expert forums, YouTube search suggestions, and official health organization FAQs (WHO, CDC).
  • Video Retrieval & Filtering: Collecting the top 25 videos for each keyword from YouTube, filtering for English language and durations between 1 and 6 minutes.
  • Annotation: Videos were annotated by domain experts and consumer-perspective annotators on medical information content, understandability (using the PEMAT framework), and overall recommendation.
  • Feature Extraction: Utilizing the YouTube Data API for metadata and the Google Cloud Video Intelligence API for speech-to-text transcription and medical terminology extraction (using UMLS). They also calculated cosine similarity metrics between video content (title, description, transcription) and search keywords.
  • Classification: Training and evaluating several machine learning models (Logistic Regression, SVM, Random Forest, XGBoost) to classify videos as recommended or not recommended, focusing on the relationship between medical information, understandability, and recommendation labels.

Datasets

The study used a dataset of YouTube videos related to COVID-19. The initial collection involved retrieving videos based on 40 curated keywords. After filtering, 304 videos were used for annotation and classification. The authors did not introduce a new benchmark dataset but rather curated one for their specific task.

Key Insights and Ideas

  • Beyond Popularity: YouTube's ranking system (view count, watch time) doesn't necessarily correlate with medical accuracy or understandability. This highlights the need for a more sophisticated curation approach.
  • Understandability is Key: The PEMAT framework proved crucial for assessing how well medical information is conveyed, emphasizing that even medically sound content can be ineffective if not understandable.
  • Expert vs. Consumer Agreement: While consumer and expert opinions on recommendation often aligned, expert judgment was prioritized for medical information accuracy, underscoring the value of domain expertise.
  • Feature Importance: Understandability, medical information expert score, readability of the description, number of shots, and cosine similarity between video description and keyword were identified as important features for classification.

Key Results

The study achieved promising results in classifying recommended COVID-19 videos.

  • Classification Performance: Logistic Regression and Random Forest models achieved the highest accuracy of 83.61%.
  • Important Features: The most influential features for classification included video understandability, medical information expert score, readability of the description, number of shots in the video, and the cosine similarity between the video description and the search keyword.
  • Qualitative Findings: The paper identified examples of both recommended videos (e.g., factual guidance from doctors) and not recommended videos (e.g., news with minimal medical information).

Limitations and Future Work

The authors acknowledge several limitations:

  • Data Loss Bias: A significant volume of data was lost from the initial video ID collection to the final classifier dataset, potentially introducing representativeness bias.
  • Future Directions:
    • Exploring additional video features to better delineate specific infectious disease topics.
  • Improving classification performance using deep learning models.
  • Incorporating new criteria such as content timeliness and accuracy.
  • Conducting evaluations by clinical experts and consumers through randomized experiments and observational studies.

Summary updated

BibTeX
@inproceedings{guo2023youtube,
  author = {Yawen Guo and Xiao Liu and Anjana Susarla and Rema Padman},
  title = {{YouTube Videos for Public Health Literacy? A Machine Learning Pipeline to Curate Covid-19 Videos}},
  booktitle = {MedInfo},
  year = {2023},
  doi = {10.3233/shti231067},
}
Figure