Drug Extraction from the Web: Summarizing Drug Experiences with Multi-Dimensional Topic Models

Michael J. Paul, Mark DredzeNAACL 2013

TL;DR

This paper introduces a novel multi-dimensional topic model, an extension of Factorial LDA (f-LDA), that leverages user-generated tags from online forums to better understand recreational drug trends. By jointly modeling drug type, route of administration, and aspects of drug use, their approach enables more targeted information extraction and summarization, proving useful for public health research.

What Problem Does This Paper Address?

The increasing availability of information on novel recreational drugs through online forums presents a challenge for public health researchers. Keeping up with drug trends, understanding usage patterns, and identifying potential side effects is crucial but time-consuming. Traditional topic models can reveal general themes in large text corpora, but they struggle to capture the specific, multi-faceted nature of drug experiences, which often involve interactions between the drug itself, how it's taken, and its effects or cultural context. Manually sifting through vast amounts of forum data is impractical, and existing automated methods often lack the granularity to pinpoint these specific interactions.

Methods

The core of this work is an extension of Factorial LDA (f-LDA), a multi-dimensional topic model. Standard LDA models documents with a single "topic" per word. f-LDA, however, models documents with a vector of latent variables, allowing for multiple dimensions (factors) to be considered simultaneously. In this paper, they use a three-dimensional f-LDA to jointly model:

  1. Drug Type: e.g., marijuana, salvia.
  2. Route of Administration: e.g., smoking, oral, injection.
  3. Aspect: e.g., effects, chemistry, health, culture.

The novelty lies in how they incorporate prior knowledge into f-LDA. Instead of a purely unsupervised approach, they leverage user-generated tags from online forums (like "Effects" or "Oral") as hierarchical priors. These priors guide the model's parameters, encouraging it to learn meaningful combinations of drug, route, and aspect. This is achieved by learning weight vectors from tagged data using a related model (SAGE) and using these as means for Gaussian priors over the f-LDA parameters. This allows the model to learn topic distributions for specific triples, like (CANNABIS, SMOKING, EFFECTS), and share parameters across related triples.

The paper also proposes using this model for extractive summarization. They extract text snippets that best represent the learned topic distributions, aiming to provide concise summaries of drug experiences.

Datasets

The primary dataset used is from drugs-forum.com, a large online forum with over 100,000 members discussing recreational drugs. They collected messages from various drug-specific subforums and utilized user-assigned tags. For evaluation, they focused on five specific drugs (mephedrone, MDPV, Bromo-Dragonfly, Spice/K2, and salvia divinorum) and used technical reports from the EU Psychonaut Project as reference summaries.

Key Insights and Ideas

  • Prior Knowledge is Key for Granularity: Purely unsupervised topic models struggle to discover specific, domain-relevant factors. Incorporating user-generated tags as priors significantly improves the model's ability to capture nuanced drug-related information.
  • Joint Modeling of Factors is Powerful: Modeling drug type, route, and aspect together (as triples) leads to a richer understanding than modeling them in isolation. This allows for insights into how routes of administration affect drug experiences, for example.
  • f-LDA as an Exploratory Tool: The model can serve as an effective tool for researchers to quickly explore large, unstructured datasets like online forums, directing them to relevant information that might otherwise be missed.
  • Extractive Summarization Benefits from Targeted Topics: Extracting snippets based on fine-grained, multi-dimensional topics yields more informative summaries than those derived from simpler models or random selection.

Key Results

  • Improved Summarization Quality: Human evaluations showed that snippets generated by their f-LDA model were rated significantly higher than those from a baseline unigram language model and random selection. The f-LDA model achieved an average score of 2.79-2.81, compared to 2.55 for the baseline.
  • Better Recall for Triples: Automatic evaluation using ROUGE scores indicated that f-LDA achieved better recall, particularly for snippets matched to triples, suggesting it captures more relevant content.
  • Expert Annotator Agreement: Expert annotators (psychiatry and behavioral pharmacology faculty) also found the f-LDA snippets to be useful, with the best f-LDA system scoring 2.57 compared to a baseline of 2.45.
  • Data Size Matters (for Triples): f-LDA-2, which used a larger dataset, performed better for triples, while f-LDA-1 (using tagged data) was slightly better for pairs.

Limitations and Future Work

  • Tag Sparsity and Quality: The model's effectiveness relies on the availability and quality of user-generated tags. Not all messages are tagged, and tags can be general or infrequent.
  • Generalization to Other Domains: While demonstrated for drug forums, the framework for incorporating priors into f-LDA could be applied to other domains with similar metadata.
  • Computational Cost: Multi-dimensional topic models can be computationally intensive, though the paper outlines optimization strategies.
  • Future Work: The authors suggest further exploration of how to incorporate more complex prior structures and applying the model to other tasks beyond summarization, such as trend detection or anomaly identification. They also note that while their model is effective, it's not a complete summarization system, but rather a tool to guide researchers.

Summary updated

BibTeX
@inproceedings{paul2013drug,
  author = {Michael J. Paul and Mark Dredze},
  title = {{Drug Extraction from the Web: Summarizing Drug Experiences with Multi-Dimensional Topic Models}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2013},
}
Figure