Discourse Mode Categorization of Bengali Social Media Health Text

Salim SazzedACL 2023

TL;DR

This paper introduces BanglaSocialHealth, the first annotated social media health corpus for Bengali, a low-resource language. They categorize health-related social media posts into four discourse modes (narrative, informative, suggestive, inquiring) and demonstrate that while transformer models like M-BERT offer a slight edge, classical ML classifiers perform comparably on this limited dataset. This work is crucial for developing NLP tools for Bengali health information.

What Problem Does This Paper Address?

Natural Language Processing (NLP) research often struggles with low-resource languages, and Bengali is a prime example. The health and medical domains are particularly data-scarce, hindering the development of useful NLP applications. While English has a wealth of annotated health data, Bengali users are increasingly turning to social media for health information and support. This gap means that valuable insights and potential interventions are missed due to the lack of tools to process this data effectively. The authors aim to bridge this gap by creating a foundational dataset and exploring classification methods for Bengali social media health text.

Methods

The core of this work involves creating a new annotated dataset and then evaluating classification models.

  • Dataset Creation: The authors manually collected health-related posts from Bengali Facebook pages. They then tokenized these posts into sentences and annotated each sentence with one of four discourse modes:
  • Narrative (NAR): Describing personal experiences, stories, or events related to health.
  • Inquiring (INQ): Posing questions or seeking information/recommendations.
  • Informative (INF): Conveying factual information about health, diseases, or treatments.
  • Suggestive (SUG): Offering advice, suggestions, or recommendations. Annotation was performed by two annotators, with a third resolving disagreements, achieving a Cohen's kappa of 0.80.

  • Classification Models: They evaluated two types of models for discourse mode classification:

  • Classical Machine Learning (CML): Logistic Regression (LR), Support Vector Machine (SVM), Random Forest (RF), and Gradient Boosting Tree (GBT) were used. These models employed word unigrams and bigrams as features, with tf-idf weighting.
  • Transformer-based Model: They fine-tuned a multilingual BERT model (M-BERT) for this task. Specifically, they kept the initial layers of M-BERT unchanged and fine-tuned the last layer for classification.

Datasets

The paper introduces the BanglaSocialHealth corpus.

  • Source: Publicly available Bengali health-related texts from social media health forums (Facebook pages).
  • Size: Approximately 2000 manually annotated sentences.
  • Annotation: Sentences are labeled with one of four discourse modes: Narrative (NAR), Informative (INF), Suggestive (SUG), and Inquiring (INQ).
  • Availability: The dataset is made publicly available for researchers.

Key Insights and Ideas

  • Low-Resource Language Challenge: The scarcity of annotated data in Bengali health domains is a significant hurdle for NLP development, highlighting the need for resources like BanglaSocialHealth.
  • Discourse Modes as Valuable Features: Categorizing health text by discourse mode offers a nuanced understanding beyond simple topic classification, enabling more targeted downstream applications like question answering or chatbot development.
  • Performance Trade-offs: While transformer models like M-BERT show promise, their advantage over classical ML classifiers is not substantial on this relatively small dataset. This suggests that for low-resource languages with limited data, simpler models can still be highly effective.
  • Class Imbalance Impact: The performance of all classifiers is significantly influenced by class distribution. The most prevalent mode (NAR) consistently shows better results, underscoring the challenge of accurately classifying underrepresented modes.

Key Results

  • Dataset Statistics: The BanglaSocialHealth corpus contains 2000 sentences. The distribution of discourse modes is imbalanced, with Narrative (NAR) being the most prevalent (42%), followed by Informative (INF) (21.20%), Suggestive (SUG) (20.36%), and Inquiring (INQ) (14.93%).
  • Classifier Performance:
    • Among classical ML classifiers, Logistic Regression (LR) achieved the best performance with an F1 score of approximately 0.78.
  • The transformer-based M-BERT model achieved a slightly higher overall F1 score of 0.80.
  • The paper notes that the improvement of M-BERT over CML classifiers was "not significant," likely due to the limited size of the labeled dataset.
  • All classifiers performed best on the NAR mode due to its higher frequency in the dataset.

Limitations and Future Work

  • Limitations:
    • Dataset Size: The corpus, while a significant contribution, is relatively small (2000 sentences), which limits the performance gains of complex models like transformers.
  • Class Imbalance: The uneven distribution of discourse modes poses a challenge for accurate classification of less frequent categories.
  • Language Nuances: The paper mentions that social media data can be noisy, with mixed languages (Bengali, English, transliterated Bengali), and sometimes missing delimiters, which required manual review.
  • Future Work:
    • Enlarging the Corpus: The authors plan to expand the size of the annotated corpus.
  • Cross-lingual Transfer: Investigating how to leverage cross-lingual resources from languages like English to improve classification performance in Bengali.
  • Downstream Tasks: Exploring the application of this corpus and classification models for practical tasks such as question-answering, misinformation detection, and information retrieval.

Summary updated

BibTeX
@inproceedings{sazzed2023discourse,
  author = {Salim Sazzed},
  title = {{Discourse Mode Categorization of Bengali Social Media Health Text}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure