TL;DR
This paper introduces MYTHTRIAGE, a novel, cost-effective pipeline for detecting opioid use disorder (OUD) myths on YouTube. By combining a lightweight model with a powerful LLM, MYTHTRIAGE significantly reduces annotation time and cost, enabling the first large-scale analysis of OUD myth prevalence on the platform.
What Problem Does This Paper Address?
Misinformation about health topics, especially high-stakes ones like opioid use disorder (OUD), is a significant public health concern. Video-sharing platforms like YouTube are major sources of health information, but also conduits for myths that can fuel treatment hesitancy and stigma. Detecting and measuring this misinformation at scale is challenging due to the sheer volume of content and the need for expert annotation. While Large Language Models (LLMs) show promise, their high computational and API costs limit their widespread use for large-scale analysis. This paper addresses the need for a scalable, efficient, and cost-effective method to detect OUD myths on YouTube.
Methods
The core of this work is MYTHTRIAGE, a triage pipeline inspired by model cascade architectures. It uses a lightweight, distilled model (DeBERTa-v3-base) to handle "easy" cases and defers more challenging ones to a powerful, but costly, LLM (GPT-40). The decision to defer is based on strategies like Maximum Softmax Probability (MSP) and Validation Error Tendencies (VET), which estimate model confidence and performance on specific classes. This approach aims to balance accuracy with cost and efficiency. The authors also employed LLMs for data augmentation to address class imbalance in their datasets.
Datasets
- OUD Search Dataset: Approximately 2.9K search results (1.8K unique videos) from 73 trending queries across 8 opioid and treatment topics.
- OUD Recommendation Dataset: Approximately 343K recommendations (164K unique videos) collected from the OUD Search Dataset, going five levels deep.
- Expert-Labeled Gold Standard Dataset: A subset of 310 videos from the OUD Search Dataset, meticulously labeled by clinical experts for 8 OUD-related myths. This dataset is crucial for training and evaluating models.
Key Insights and Ideas
- The "Triage" Approach is Key: MYTHTRIAGE's success hinges on intelligently routing tasks. By using a lightweight model for the majority of cases and only engaging the expensive LLM for difficult ones, they achieve significant cost and time savings.
- LLMs as "Teachers" for Distillation: The paper demonstrates the effectiveness of using a high-performing LLM (GPT-40) to generate synthetic labels for training smaller, more efficient models. This is a powerful strategy for scaling annotation when expert labels are scarce.
- YouTube's Search Filters Matter: The study found that YouTube's default "Relevance" search filter tends to surface more myth-opposing content, while alternative filters like "Upload Date," "View Count," and "Rating" shift the bias towards more myth-supporting content. This has implications for platform moderation.
- Myth Persistence in Recommendations: Even when a video opposes a myth, its recommendations can still lead users down a rabbit hole of myth-supporting content, highlighting a subtle but persistent pathway for misinformation spread.
Key Results
- MYTHTRIAGE achieved up to 0.86 macro F1-score on myth detection, comparable to expert labeling.
- It estimated to reduce annotation time and financial cost by over 76% compared to full LLM labeling and over 96% compared to expert labeling.
- Nearly 20% of YouTube search results for OUD-related queries supported myths.
- Videos related to Kratom showed the highest prevalence of myths, with 36% of search results supporting myths.
- At the first level of recommendations, 4.9% of videos supported myths, with a trend of increasing exposure to myth-supporting content at deeper recommendation levels.
Limitations and Future Work
- Focus on OUD: The study is specific to OUD myths, and future work could extend the methodology to other health domains like COVID-19 or mental health.
- Limited Myth Set: While 8 myths were examined, many other OUD-related myths and topics remain underexplored.
- YouTube-Centric: The analysis is primarily on YouTube, and future work could explore other platforms like TikTok or BitChute.
- Language and Context: The study focused on English queries and a US context, limiting generalizability to other languages and regions.
- Model Misclassification: Despite extensive validation, LLMs can still misclassify content, and the distillation process might introduce some error.
- Future Work: The authors suggest expanding the myth set, exploring deeper personalization effects on user exposure, and extending the analysis to other platforms. They also propose adapting MYTHTRIAGE to use smaller, open-weight LLMs for potentially better performance and cost-effectiveness.
Summary updated