TL;DR
This paper introduces deep learning models for automatically identifying narratives in online health communities, outperforming previous methods. By analyzing the features these models rely on, the research offers insights into what makes a post "narrative" and how these models can generalize across different health organizations.
What Problem Does This Paper Address?
Online health communities are vibrant spaces where individuals share personal experiences, seek support, and disseminate health information. Narratives, or stories about events, are particularly effective in communicating health risks and promoting behavior change. However, manually identifying these narratives within the vast amount of social media data is a time-consuming and expensive process, hindering large-scale research. This paper aims to bridge this gap by developing and evaluating automatic narrative detection methods, specifically leveraging modern deep learning techniques, to make studying narratives in online health contexts more scalable and efficient.
Methods
The authors explore various text classification approaches for narrative detection. They compare classical machine learning models (Naive Bayes, Logistic Regression, SVM) using bag-of-words features with transformer-based deep learning models, including DistilBERT, BERT, and RoBERTa. These deep learning models are fine-tuned for the narrative detection task. To understand why these models are effective, they employ the LIME (Local Interpretable Model-Agnostic Explanations) technique to analyze feature importance, identifying which words and phrases contribute most to classifying a post as narrative or non-narrative. They also use Global Aggregations of Local Explanations (GALE) to aggregate these importance scores across the dataset.
Datasets
The study utilizes a newly annotated dataset of social media posts from five breast cancer non-profit organizations. Posts were collected from 2016 to 2021. The top 10% of posts by total interactions were sampled and annotated by two expert coders, with disagreements resolved through discussion. The final annotated dataset contained 849 posts with an overall narrative presence of 49.0%.
Key Insights and Ideas
- Deep Learning Dominance: Transformer-based deep learning models significantly outperform traditional machine learning methods for narrative detection in this domain. This suggests that the complex contextual understanding offered by these models is crucial for identifying nuanced narrative structures.
- Generalizability is Achievable: The deep learning models demonstrate good generalization capabilities, meaning they can effectively detect narratives even when trained on data from some organizations and tested on data from others. This is a critical finding for real-world deployment.
- Personalization is Key: Features like personal pronouns (e.g., "she," "her") and first names are strong indicators of narrative content. This aligns with the intuitive understanding that narratives are about personal experiences.
- Non-Narrative Clues: Conversely, terms related to scientific studies, general descriptions, and engagement-seeking hashtags are often associated with non-narrative posts. This helps the models distinguish between personal stories and informational or promotional content.
- Model Differences: While both BERT and SVM identify personal pronouns and names as important, BERT places a greater emphasis on "she" and first names, while SVM gives more weight to "her." This highlights subtle differences in how these models learn to interpret narrative cues.
Key Results
- The BERT model achieved the best overall performance, with an F1 score of 0.988, significantly outperforming classical SVM models (F1 score of 0.901) and the previous state-of-the-art SVM-trigram model.
- When testing generalization by holding out one organization's data, the BERT model consistently achieved high F1 scores (e.g., 0.991 for Breast Cancer Now), indicating strong cross-organizational performance.
- When training on data from only one organization and testing on the rest, performance varied, with models trained on more diverse data (like NBCF Australia) showing better generalization.
- Feature analysis revealed that personal pronouns and first names were highly important for narrative detection. For non-narrative detection, terms related to scientific studies and general descriptions were key.
Limitations and Future Work
The authors acknowledge that deep learning models are sensitive to dataset quality and can be susceptible to annotator and dataset bias. They suggest that future work could explore more advanced feature engineering or different model architectures to further improve performance and robustness. Additionally, investigating the impact of different types of narratives (e.g., personal testimonials vs. fictional stories) and their specific linguistic markers could be a valuable next step. The work also opens avenues for larger-scale studies on the impact of narratives in health conversations.
Summary updated