Using PPM for Health Related Text Detection

Victoria Bobicev, Victoria Lazu, Daniela IstratiEMNLP 2018

TL;DR

This paper explores the use of Prediction by Partial Matching (PPM), a text compression technique, for classifying social media posts related to drug intake and vaccination behavior. While achieving competitive results, their approach highlights the challenges of applying standard methods to noisy, real-world social media data and suggests avenues for improvement.

What Problem Does This Paper Address?

The paper tackles the challenge of automatically identifying health-related information within the vast and often noisy landscape of social media. Specifically, it focuses on two tasks from the SMM4H (Social Media Mining for Health Applications) challenge: detecting posts mentioning drug names and posts discussing vaccination behavior. The inherent messiness of social media text, including misspellings, slang, and abbreviations, makes traditional natural language processing (NLP) techniques struggle. This work investigates whether a compression-based approach, PPM, can effectively handle this noise and accurately classify these health-related posts.

Methods

The core of this paper's methodology is the application of Prediction by Partial Matching (PPM). PPM is an adaptive, finite-context method originally developed for text compression. It works by predicting the next character in a sequence based on the probabilities derived from preceding characters within a defined context window. The paper uses a character-based PPM, treating text as a string of characters.

A key aspect of their approach is the blending strategy for combining predictions from different context lengths. They assign weights to probabilities from various context models and calculate a weighted sum. The paper notes that a maximal context length of 5 was found to be optimal for text compression and was adopted here.

For classification, they leverage the concept of cross-entropy. Cross-entropy measures the similarity between two texts by calculating the entropy of one text using a model trained on another. In this context, they trained separate PPM models for each class (e.g., posts mentioning drugs vs. posts not mentioning drugs). Then, for an unseen tweet, they calculated its cross-entropy against each class model. The class with the lowest cross-entropy was assigned to the tweet.

To potentially improve performance, they also explored feature selection using Gain Ratio. This technique helps identify the most informative words (features) in the text for classification, allowing them to remove less relevant words before building the PPM models.

Datasets

The study utilized data from the SMM4H challenge.

  • Task 1 (Drug Name Detection):
    • Training data: 9130 annotated tweets (4730 mentioning drug names, 4400 not).
  • Evaluation set: 5384 tweets.
  • Task 4 (Vaccination Behavior Detection):
    • Training data: 6941 annotated tweets (1979 mentioning vaccination behavior, 4962 not).
  • Evaluation set: 161 tweets.

The data was sourced from Twitter and provided by the challenge organizers.

Key Insights and Ideas

  • Compression as a Classification Tool: The paper demonstrates that techniques rooted in text compression, like PPM, can be adapted for text classification tasks, particularly in noisy domains like social media.
  • Cross-Entropy for Similarity: The core idea of using cross-entropy as a similarity metric between a tweet and class-specific PPM models is a clever way to leverage the statistical language modeling capabilities of PPM for classification.
  • Character-Level vs. Word-Level: The authors note that character-level PPM models tend to perform better on noisy texts with misspellings and slang, which is a common characteristic of social media data.
  • Feature Selection Matters: Employing Gain Ratio to select important words before model creation led to improved classification results, highlighting the value of preprocessing and feature engineering.

Key Results

  • Task 1 (Drug Name Detection):
    • Initial 10-fold cross-validation on Task 4 data yielded: Precision = 0.839, Recall = 0.838, F-score = 0.839.
  • After feature selection using Gain Ratio, results improved to: Precision = 0.861, Recall = 0.858, F-score = 0.859.
  • Final blind test set results: Precision = 0.841, Recall = 0.860, F-score = 0.850.
  • Comparison: The mean result for all participating teams was P=0.890, R=0.872, F=0.880.
  • Task 4 (Vaccination Behavior Detection):
    • Initial results: Precision = 0.842, Recall = 0.814, F-score = 0.828.
  • Final blind test set results: Precision = 0.829, Recall = 0.808, F-score = 0.818.
  • Comparison: The mean result for all participated teams was P=0.826, R=0.858, F=0.840.

Overall, their results were competitive but slightly lower than the mean performance of other teams in both tasks.

Limitations and Future Work

  • Accuracy Gap: The authors acknowledge that their results are lower than the average performance achieved by other teams in both tasks.
  • Suitability of PPM: They question whether PPM is the most suitable method for this specific type of text classification.
  • Preprocessing: They suggest that more extensive preprocessing of the texts might be necessary before applying classification models.
  • Term Weighting: The current approach treats all terms uniformly. They propose exploring term weighting techniques to give more importance to certain words during calculations.
  • Future Plans: The authors plan to implement more sophisticated preprocessing and term weighting strategies for future challenges.

Summary updated

BibTeX
@inproceedings{bobicev2018using,
  author = {Victoria Bobicev and Victoria Lazu and Daniela Istrati},
  title = {{Using PPM for Health Related Text Detection}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
}
Figure