TL;DR
This paper tackles the challenge of automatically identifying and categorizing treatments discussed in online autism communities to understand real-world treatment adoption. By developing a machine learning approach that jointly identifies treatment mentions and their attributions (who is using or discussing them), the researchers create a valuable catalogue of treatments used by individuals with Autism Spectrum Disorder (ASD). This work lays the groundwork for comparing online discussions with clinical guidelines and understanding patient treatment journeys.
What Problem Does This Paper Address?
Online health communities (OHCs) are rich sources of information and support for patients and caregivers, particularly for chronic conditions like Autism Spectrum Disorder (ASD). While these communities facilitate discussions about treatments, a key research question remains: do treatments discussed online actually get used by patients in their daily lives? This is crucial for understanding treatment adherence, patient decision-making, and the gap between clinical recommendations and real-world practice. Traditional methods for gathering this information are often difficult and expensive. This paper aims to bridge this gap by automatically extracting and classifying treatment information from OHCs, specifically focusing on autism communities where parents discuss their children's care. A unique challenge in this domain is distinguishing between treatments discussed generally and those actually adopted by patients, especially since the primary users of autism forums are caregivers.
Methods
The core of this paper's methodology lies in a two-step process using machine learning:
- Treatment Mention Identification: This involves recognizing specific terms that refer to treatments within the text of forum posts. This is framed as a Named Entity Recognition (NER) task.
- Treatment Attribution Identification: Once a treatment is identified, the system determines who is associated with that treatment. The authors define five attribution labels:
pt: Mentions indicating actual usage or usage history by patients (ASD children).pt-gen: Mentions tied to the patient but not indicating actual usage.cg: Mentions tied to the caregiver (usually the parent posting).others: Mentions tied to individuals other than the caregiver or patient.gen: Mentions not tied to a specific individual (general discussions).
The authors employ a Conditional Random Fields (CRF) model to jointly perform both mention identification and attribution classification. This approach allows the model to learn dependencies between identifying a treatment and assigning its attribution simultaneously, which is more effective than separate steps. They experiment with various feature sets, including lexical (words, lemmas, POS tags), semantic (using lexicons like WordNet and UMLS), and syntactic features.
Datasets
The study utilizes a dataset collected from two publicly available autism forums: autismweb.com and autism-pdd.net. The data was crawled in March 2015. The combined dataset includes sub-forums, threads, posts, and author information. A subset of 500 posts was manually annotated for treatment mentions and their attributions to train and evaluate the models.
Key Insights and Ideas
- Distinguishing Discussion from Usage is Key: Simply finding treatment names in online discussions isn't enough. The crucial insight is the need to differentiate between general mentions and those indicating actual patient usage, especially in caregiver-dominated forums.
- Joint Modeling for Better Performance: Performing treatment mention identification and attribution classification together using a CRF model significantly outperforms separate approaches, highlighting the interconnectedness of these tasks.
- Lexical Features are Sufficient for Mention Identification: Surprisingly, for identifying treatment mentions, basic lexical features alone proved to be highly effective, suggesting that the specific words used are strong indicators.
- Syntactic Features Aid Attribution: While not crucial for mention identification, syntactic features play a more significant role in accurately assigning attributions, likely by providing broader sentence context.
- "pt" and "gen" are Dominant: The most frequent attribution labels in the dataset are "patient usage" (
pt) and "general discussion" (gen), which aligns with the nature of these forums. - Real-World Usage Differs from Popularity: The study reveals that treatments frequently discussed (e.g., chelation) are not always the most commonly used by patients. Conversely, less frequently discussed but established interventions (e.g., speech therapy) show higher actual usage.
Key Results
- The CRF-based system achieved strong performance in treatment mention identification, outperforming a dictionary-based baseline. For example, using lexical features, the F-score for mention identification was 82.9%.
- In the end-to-end evaluation (joint mention and attribution), the CRF model with lexical, semantic, and syntactic features achieved an F-score of 66.8% for the
ptattribution. - When simplifying the task to a binary classification (patient usage vs. non-patient usage), the accuracy for identifying patient usage (
pt) improved by about 4-5%. - The study identified over 164,000 mentions of nearly 4,000 distinct treatment terms across the entire dataset.
- The top treatments by actual usage (
ptattribution) included probiotics, speech therapy, and early intervention, while chelation, though frequently discussed, ranked lower in actual usage.
Limitations and Future Work
- Annotation Bias: The authors acknowledge that the annotation schema, while aiming for local classification, might miss nuances that context could reveal. They also note that co-references (pronouns) were ignored, which could affect results if not handled carefully.
- Questionable Attributions: Some identified attributions, particularly for caregivers (
cg), were questionable and might be due to classification errors. - Temporal Analysis: While the paper mentions investigating temporal information, the detailed findings on how perceptions and usage change over time are presented as a starting point for future work.
- Comparison with Clinical Guidelines: The authors emphasize that a crucial next step is to compare the automatically generated treatment catalogues with established clinical guidelines to quantify the gap between recommended and actual practices.
- Understanding Community Influence: Further research is needed to explore how sustained participation in online communities influences members' treatment decisions and how this impacts their psychological and physical well-being.
Summary updated