Mining Social Media for Barriers to Opioid Recovery with LLMs

Vinu Ekanayake, Md Sultan Al Nahian, Ramakanth KavuluruNAACL 2025

TL;DR

This paper introduces a novel, semi-automated approach using Large Language Models (LLMs) to mine social media for barriers to opioid recovery. By analyzing Reddit posts, the study identifies both established and emerging challenges, revealing how the COVID-19 pandemic significantly shifted the landscape of these barriers, emphasizing systemic issues over personal ones.

What Problem Does This Paper Address?

The opioid epidemic remains a critical public health crisis, with individual, social, and structural barriers significantly hindering recovery. Traditional methods for identifying these barriers, such as focus groups and surveys, are expensive, time-consuming, and struggle to capture the dynamic evolution of new challenges. While social media platforms like Reddit offer a rich source of user-generated content, manually sifting through vast amounts of data to identify specific barriers is a daunting task. This paper addresses the need for an automated and scalable method to extract, classify, and track these barriers, particularly to understand how external events like the COVID-19 pandemic impact recovery.

Methods

The authors developed a multi-stage, semi-automated pipeline leveraging LLMs, specifically GPT-4, to address this challenge.

  • Literature-Derived Barrier Curation: A manual literature review identified 21 established barriers to opioid recovery.
  • Barrier Expression Extraction: A three-stage prompting strategy with GPT-4 was employed to extract barrier expressions from Reddit posts. This involved an initial extraction, a verification step to ensure adherence to guidelines (user's own experience, explicit mention of relapse risk), and a finalization step to refine the list.
  • Mapping to Literature-Derived Barriers (LDBs): Extracted Reddit barriers were converted into embeddings and compared to the LDBs using cosine similarity. Barriers with high similarity were mapped to existing LDBs, while those with low similarity were classified as "new."
  • Clustering of New Barriers: "New" barriers were further processed using agglomerative hierarchical clustering on their embeddings to group similar expressions and identify coherent "emerging" barriers. A secondary clustering step, incorporating key phrases generated by GPT-4, refined these clusters for better interpretability.
  • Temporal Analysis: The prevalence of both LDBs and emerging barriers was analyzed across pre-pandemic (January 2018–March 2020) and pandemic (March 2020–December 2021) periods to identify shifts.

Datasets

  • r/OpiatesRecovery Subreddit Data: Posts collected from January 1, 2018, to December 31, 2021. After filtering for posts with at least 50 words, the dataset comprised 14,735 posts from 7,202 unique users.
  • Literature-Derived Barriers (LDBs): A curated list of 21 barriers identified through a manual review of scientific literature.

Key Insights and Ideas

  • LLMs as a Scalable Tool for Qualitative Research: The study demonstrates the potential of LLMs to automate and scale qualitative data analysis, complementing traditional methods and providing a more dynamic understanding of complex issues like addiction recovery.
  • The Shifting Landscape of Recovery Barriers: The COVID-19 pandemic significantly altered the prominence of recovery barriers. While some personal and social barriers (like stigma) appeared to decrease, systemic obstacles (like treatment discontinuity and exclusionary practices) saw a notable rise.
  • Emerging Barriers Highlighted by the Pandemic: The pandemic brought to light new challenges such as increased isolation due to work-from-home policies and disrupted support group sessions, underscoring the vulnerability of recovery to external crises.
  • Nuance in Social Media Data: The study acknowledges the inherent complexity and semantic nuances of user-generated text, which can make precise classification and clustering challenging, even with advanced NLP techniques.

Key Results

  • High Performance of CoT + Verification Prompting: The multi-stage prompting strategy, specifically "Chain-of-Thought with Verification," achieved high precision (95.14%) and recall (94.24%) in extracting barrier expressions.
  • Significant Proportion of Novel Barriers: Out of 17,603 extracted Reddit barriers, 12,038 (40.7%) were classified as "new," indicating the emergence of challenges not well-documented in prior literature.
  • Temporal Shifts:
    • Decreases: Identity difficulties (ID 14) saw the biggest decline (-36.87%), followed by secrecy/fear about the past (ID 9, -29.04%) and over-reliance on others (ID 7, -21.28%). Fear of stigma (ID 2) also dropped (-17.34%).
  • Increases: Lack of connection between emergency care and professional medical treatment (ID 16) soared by 128.39%. Poor staff attitudes/training (ID 15) rose by 84.997%, and unsuitable/ineffective services (ID 4) surged by 71.29%.
  • Emerging Barriers During Pandemic: Several emerging barriers rose substantially, particularly those related to pandemic-induced isolation, reduced professional support, and disruption of recovery resources like support groups.

Limitations and Future Work

  • Human Intervention Required: The approach, while semi-automated, still requires significant human steering for prompt engineering, manual annotation for evaluation, and determining thresholds for mapping and clustering.
  • Subtle Nuances and Context: Subtle linguistic nuances and context-dependent barriers might still be misclassified or overlooked by the LLM.
  • Self-Reported Data Caveats: Reddit data carries intrinsic caveats, including potential exaggeration, underreporting, or skewed user demographics.
  • Future Work: The authors suggest exploring more advanced LLMs (e.g., Gemini, GPT-4o) to further automate LDB curation and barrier extraction. Dynamic topic models could also enhance clustering by incorporating topic distribution. The method is proposed to be generalizable to other substance use disorders (alcohol, stimulants) by adapting domain-specific language.

Summary updated

BibTeX
@inproceedings{ekanayake2025mining,
  author = {Vinu Ekanayake and Md Sultan Al Nahian and Ramakanth Kavuluru},
  title = {{Mining Social Media for Barriers to Opioid Recovery with LLMs}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
}
Figure