LHS712NV at #SMM4H 2024 Task 4: Using BERT to classify Reddit posts on non-medical substance use

Valeria Fraga, Neha Nair, Dalton Simancek, V.G.Vinod VydiswaranACL 2024

TL;DR

This paper details the use of BERT, a powerful language model, to identify clinical and social impacts of non-medical substance use in Reddit posts for the #SMM4H 2024 shared task. While BERT showed promise in recognizing clinical impacts, it struggled with social impacts, highlighting the ongoing challenge of nuanced entity recognition in social media health data.

What Problem Does This Paper Address?

Understanding the multifaceted consequences of non-medical substance use is critical for public health. This includes not only direct health effects (clinical impacts) but also broader societal ramifications (social impacts). Social media platforms like Reddit are rich sources of user-generated content that can offer real-time insights into these impacts. However, extracting this information at scale requires sophisticated natural language processing (NLP) techniques. The #SMM4H 2024 Task 4 specifically aimed to address this by challenging participants to develop systems capable of identifying "Clinical Impacts" and "Social Impacts" entities within Reddit posts related to non-medical substance use.

Methods

The core of this work relies on the Bidirectional Encoder Representations from Transformers (BERT) model. BERT is a state-of-the-art deep learning model renowned for its ability to understand the contextual meaning of words in text. The authors employed a standard approach of fine-tuning a pre-trained BERT model for the specific task of Named Entity Recognition (NER).

The process involved:

  1. Data Preprocessing: The Reddit posts were tokenized, and each token was assigned one of three labels: "Clinical Impacts," "Social Impacts," or "No Label."
  2. BERT Fine-tuning: A pre-trained BERT model was fine-tuned on the labeled Reddit data using the Hugging Face Transformers library. Standard training parameters like batch size, learning rate, and the number of epochs were configured.
  3. Evaluation: The model's performance was assessed using precision, recall, F1 score, and accuracy metrics on both training and validation datasets. A confusion matrix was generated to visualize classification performance.

Datasets

The primary dataset used in this study was derived from Reddit posts as part of the #SMM4H 2024 Task 4 shared task. The paper mentions that the training dataset consisted of approximately 800 posts. The dataset was pre-anonymized.

Key Insights and Ideas

  • BERT's Strength in Clinical Impact Recognition: The model demonstrated a strong ability to identify clinical impacts, suggesting that these types of entities are more clearly and consistently expressed in the data.
  • The Challenge of Social Impact Recognition: A significant finding is the model's poor performance in identifying "Social Impacts." This highlights the inherent difficulty in defining and extracting these more abstract and context-dependent entities from user-generated text. Social impacts can be more subtle, varied, and less explicitly stated than clinical ones.
  • Data Imbalance and Majority Class Dominance: The confusion matrix and accuracy percentages (Table 3) reveal that the model heavily favored the "No Label" class. This is a common issue in NER tasks, especially when dealing with imbalanced datasets where the majority class (no relevant entity) significantly outweighs the minority classes.

Key Results

The paper reports the following key quantitative results:

  • Validation Data: The model achieved a token-level F1 score of 0.892 on the validation data.
  • Test Data: The model's performance dropped significantly on the test data, with a token-level F1 score of 17.15% and a relaxed F1 score of 19.12%.
  • Accuracy on Labels (Validation):
    • No label: 99.5%
    • Clinical Impacts: 5.9%
    • Social Impacts: 0.0%

These results indicate that while the model was generally good at identifying posts without clinical or social impacts, its ability to pinpoint actual instances of these impacts, particularly social ones, was very limited on the test set.

Limitations and Future Work

The authors acknowledge several limitations:

  • Data Unpredictability: Reddit data is dynamic and unregulated, potentially introducing noise and biases that affect generalizability.
  • Limited Training Data: The dataset of approximately 800 posts might not be sufficient to capture the full spectrum of patterns and nuances, limiting the model's ability to discern complex relationships.
  • Model Stability: Fluctuating F1 scores (ranging from 0.4 to 0.9) suggest potential issues with model stability or sensitivity to varying conditions.
  • Exclusion of "Entity or Not" Column: The authors did not use an "entity or not" column in their training, which could have provided additional context. They also did not investigate other models, which could have offered comparative insights.

Future work could involve:

  • Addressing the identified constraints to enhance validity and applicability.
  • Exploring the performance of other NLP models on the same data.
  • Developing more robust methodologies and data validation techniques.
  • Further refinement to improve the accurate identification of "Social Impact" instances.

Summary updated

BibTeX
@inproceedings{fraga2024lhs,
  author = {Valeria Fraga and Neha Nair and Dalton Simancek and V.G.Vinod Vydiswaran},
  title = {{LHS712NV at \#SMM4H 2024 Task 4: Using BERT to classify Reddit posts on non-medical substance use}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure