Identifying professions & occupations in Health-related Social Media using Natural Language Processing

Alberto Mesa Murgado, Ana Parras Portillo, Pilar López Úbeda, Maite Martin, Alfonso Ureña-LópezNAACL 2021

TL;DR

This paper tackles the challenge of automatically identifying professions and occupations within health-related social media posts using Natural Language Processing (NLP). The authors developed systems for both classifying tweets that mention professions and for extracting and tagging these mentions, achieving competitive results in the SMM4H 2021 ProfNER shared task. This work is crucial for public health monitoring, enabling better understanding of at-risk professional sectors during health crises.

What Problem Does This Paper Address?

The paper addresses the growing need to extract valuable information from the vast amount of health-related content shared on social media. Specifically, it focuses on identifying mentions of professions and occupations within these posts. This information is vital for public health initiatives, such as understanding which professional groups are most vulnerable to health threats like the COVID-19 pandemic or are experiencing increased mental health strain due to current events. The challenge lies in the unstructured nature of social media text and the need for accurate and efficient NLP techniques to perform this identification.

Methods

The researchers employed a combination of established NLP techniques and custom-built resources. For the binary classification task (determining if a tweet mentions a profession), they utilized two approaches: a Support Vector Machine (SVM) and a bidirectional Long Short-Term Memory (BiLSTM) recurrent neural network. Both were enhanced by incorporating a custom dictionary of 776 professions.

For the Named Entity Recognition (NER) task (detecting and classifying profession mentions), they opted for a Conditional Random Fields (CRF) approach. This method was also augmented with their custom dictionaries of professions and working statuses, along with features derived from surrounding words. The CRF model was trained using the L-BFGS gradient descent method.

Datasets

The study utilized data from the SMM4H 2021 ProfNER Shared Task. This included:

  • A training set of 6,000 tweets.
  • A development (dev) set of 2,000 tweets for validation.
  • A test set of 2,000 processed tweets for final evaluation.
  • A background set of 25,000 raw tweets.

The dataset provided tweet identifiers, text spans, a binary label indicating the presence of a profession, and annotated entities in the Inside-Outside-Beginning (IOB) format.

Key Insights and Ideas

  • The Power of Dictionaries: The researchers found that augmenting standard NLP models (SVM, BiLSTM, CRF) with custom-built dictionaries of professions and working statuses significantly improved performance. This highlights the value of domain-specific knowledge in NLP tasks.
  • Task-Specific Approaches: The paper demonstrates that different NLP architectures are suited for different sub-tasks. BiLSTM and SVM were used for classification, while CRF proved effective for the more granular NER task.
  • Social Media as a Health Data Source: The work reinforces the potential of social media as a rich source for public health monitoring and research, provided that effective NLP tools are developed to process the data.

Key Results

For Task 7a (Binary Classification) on the test set:

  • The SVM model achieved a precision (P) of 0.9185, recall (R) of 0.8553, and F1-score of 0.85.
  • The BiLSTM model achieved a precision (P) of 0.7612, recall (R) of 0.4752, and F1-score of 0.59.

For Task 7b (NER Offset Classification) on the test set:

  • The CRF model with dictionaries achieved a precision (P) of 0.824, recall (R) of 0.652, and F1-score of 0.728.
  • A simpler CRF model without dictionaries achieved a precision (P) of 0.852, recall (R) of 0.597, and F1-score of 0.702.

The authors noted that their binary classification approaches performed significantly below expectations on the test set, showing a 21% decrease in F1-score compared to their training results. However, their NER performance was closer to the median and showed a smaller performance drop from training to testing.

Limitations and Future Work

The authors acknowledge that their binary classification systems underperformed on the test set, falling significantly below the median. They plan to conduct a deeper analysis using the gold test data to understand the reasons behind this performance gap and identify areas for improvement. Future work will focus on refining these systems to enhance their accuracy and reliability for identifying professions and occupations in health-related social media content.

Summary updated

BibTeX
@inproceedings{murgado2021identifying,
  author = {Alberto Mesa Murgado and Ana Parras Portillo and Pilar López Úbeda and Maite Martin and Alfonso Ureña-López},
  title = {{Identifying professions \& occupations in Health-related Social Media using Natural Language Processing}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
}
Figure