Conversation-Aware Filtering of Online Patient Forum Messages

Anne Dirkson, Suzan Verberne, Wessel KraaijSMM4H @ COLING 2020 · OpenReview

Abstract

Previous approaches to NLP tasks on online patient forums have been limited to single posts as units, thereby neglecting the overarching conversational structure. In this paper we explore the benefit of exploiting conversational context for filtering posts relevant to a specific medical topic. We experiment with two approaches to add conversational context to a BERT model: a sequential CRF layer and manually engineered features. Although neither approach can outperform the F1 score of the BERT baseline, we find that adding a sequential layer improves precision for all target classes whereas adding a non-sequential layer with manually engineered features leads to a higher recall for two out of three target classes. Thus, depending on the end goal, conversation-aware modelling may be beneficial for identifying relevant messages.

Introduction

When researchers analyze online patient forums to find mentions of Adverse Drug Reactions (ADRs) or medical misinformation, the primary challenge is "relevance classification"—filtering out noise to find meaningful signals. Traditionally, this is performed on a post-by-post basis. However, human readers understand posts within their conversational context; a reply often inherits the topic of the preceding question. By ignoring this "threadiness," current models miss vital discourse-level cues. This paper investigates whether conversation-aware models—using sequential layers or manually engineered features—can better identify relevant medical information within patient threads.

Methods

The researchers compared a standard DistilBERT baseline, which analyzes posts in isolation, against two conversation-aware strategies:

  1. Sequential Modeling (CRF): A Conditional Random Field (CRF) layer was added to predict sequences, considering the likelihood of a label given the labels of surrounding posts.
  2. Manually Engineered Features: Features were extracted to describe a post's position and relationship within a thread. These include relational metrics (similarity to previous messages), positional data (distance from thread start), and label distribution (running counts of positive/negative labels).
Table of manually engineered features categorized by type (Local, Relational, Positional, Label distribution, and Discourse).

Additionally, discourse features were used to identify dialogue acts, such as whether a post was a question, explanation, or statement. These features were tested using blending-based ensembles, where DistilBERT confidence scores were fed into either the sequential CRF or a non-sequential SVM.

Datasets

The study utilized two distinct datasets:

  • Medical Misinformation Dataset: 1,566 posts from MedHelp annotated for misinformation.
  • ADR Discussions (In-house): 4,195 posts from a Facebook group for GIST (Gastro Intestinal Stromal Tumor) patients, annotated for Adverse Drug Responses (ADRs) and Coping Strategies.

Results and Insights

The Clustering Effect

A key finding was that relevant posts are not randomly distributed but tend to cluster. In the ADR dataset, while the baseline probability of a post being an ADR is 22.9%, this jumps to 40% if the preceding post was also an ADR. This confirms that conversational structure is a powerful signal for classification.

Distribution of positive labels relative to other positive labels, showing a strong peak at zero, which indicates significant clustering within threads.

Precision vs. Recall Trade-offs

While the DistilBERT baseline proved difficult to beat in terms of overall $F_1$ score, the conversation-aware models offered specific advantages:

  • Sequential CRF layers consistently improved precision across all classes.
  • Manual features in a non-sequential SVM led to higher recall for two out of three classes.

For Misinformation, the conversation-aware models actually outperformed the baseline, with an SVM using sentence vectors achieving an $F_1$ of 0.478 compared to BERT's 0.366. For ADR detection, the CRF + BERTpred model achieved a statistically significant boost in precision (0.728 vs. 0.715).

Conclusion

Conversation-aware modeling provides a valuable mechanism for tuning models toward higher precision or recall based on research needs. While BERT captures some implicit context, explicit conversational features—particularly the label of the previous post and running counts of labels—add significant value. Future work will explore these techniques in concept normalization and larger, more imbalanced datasets.

Summary updated

Abstract

Previous approaches to NLP tasks on online patient forums have been limited to single posts as units, thereby neglecting the overarching conversational structure. In this paper we explore the benefit of exploiting conversational context for filtering posts relevant to a specific medical topic. We experiment with two approaches to add conversational context to a BERT model: a sequential CRF layer and manually engineered features. Although neither approach can outperform the F1 score of the BERT baseline, we find that adding a sequential layer improves precision for all target classes whereas adding a non-sequential layer with manually engineered features leads to a higher recall for two out of three target classes. Thus, depending on the end goal, conversation-aware modelling may be beneficial for identifying relevant messages. We hope our findings encourage other researchers in this domain to move beyond studying messages in isolation towards more discourse-based data collection and classification. We release our code for the purpose of follow-up research.
BibTeX
@inproceedings{dirkson2020conversation,
  author = {Anne Dirkson and Suzan Verberne and Wessel Kraaij},
  title = {{Conversation-Aware Filtering of Online Patient Forum Messages}},
  booktitle = {SMM4H @ COLING},
  year = {2020},
  url = {https://openreview.net/forum?id=9NN2se3Iq8},
}
Figure