An Annotated Dataset and Automatic Approaches for Discourse Mode Identification in Low-resource Bengali Language

Salim SazzedNAACL 2022

TL;DR

This paper introduces the first annotated dataset for discourse mode identification in Bengali, a low-resource language. They developed both classical machine learning and transformer-based approaches, finding that a fine-tuned multilingual BERT model offers the best performance for automatically classifying sentences into narrative, descriptive, or informative modes.

What Problem Does This Paper Address?

Understanding how language is used to convey information, tell stories, or describe things (i.e., discourse modes) is crucial for many Natural Language Processing (NLP) tasks like summarization and question answering. However, research in this area has been limited for low-resource languages like Bengali. Existing work often focuses on languages with abundant data, leaving a significant gap in our ability to analyze discourse in languages spoken by millions but with fewer digital resources. This paper aims to bridge this gap by creating a Bengali discourse dataset and exploring automatic methods for identifying discourse modes within Bengali text.

Methods

The authors employed two main categories of methods for discourse mode identification:

  • Classical Machine Learning (CML): They used standard CML classifiers like Logistic Regression (LR), Support Vector Machines (SVM), Random Forest (RF), and Extra Trees (ET). These models were trained using word n-gram features (unigrams and bigrams) extracted from the text, with TF-IDF scores applied.
  • Deep Learning (Transformer-based): They leveraged the multilingual BERT (M-BERT) model, a powerful pre-trained language model that has demonstrated state-of-the-art performance across various NLP tasks. M-BERT was fine-tuned specifically for this discourse mode classification task. The fine-tuning process focused on the last layer of the M-BERT model, adding a classification layer on top.

Datasets

The core contribution is the creation of a newly annotated Bengali discourse mode corpus.

  • Source: The corpus was built from excerpts of six 20th-century Bengali novels.
  • Annotation: Sentences were manually annotated into three discourse modes: narrative, descriptive, and informative.
  • Size: The final dataset consists of 3310 sentences.
  • Annotation Process: Three native Bengali speakers with university-level education performed the annotation, using provided guidelines and considering surrounding sentences for context. They were instructed to identify the dominant mode when multiple were present.
  • Comparison: This dataset is unique as it's the first of its kind for Bengali discourse mode identification, addressing a significant lack of resources.

Key Insights and Ideas

  • Bengali Discourse is Under-Researched: The paper highlights the significant lack of NLP resources and research for Bengali, despite it being a widely spoken language.
  • Narrative Dominance: In fictional narratives like novels, the narrative mode is overwhelmingly dominant, likely because novels inherently focus on storytelling and event progression.
  • Descriptive Sentences are Longer: Descriptive sentences tend to be longer than narrative or informative ones, which makes sense as they require more detail to paint a picture for the reader.
  • Transformer Models Outperform CML: Fine-tuned transformer models like M-BERT show a clear advantage over traditional CML approaches, even with a relatively small dataset.
  • Informative Mode is Challenging: All models struggled with the informative class, likely due to its lower prevalence and potentially more subtle linguistic cues compared to narrative or descriptive modes.

Key Results

  • M-BERT Performance: The multilingual BERT model achieved the highest F1 score for the narrative class (0.912), outperforming CML classifiers. For descriptive and informative classes, M-BERT's performance was comparable to the best CML classifiers (LR and SVM).
  • CML Performance: Among CML classifiers, Logistic Regression (LR) and Support Vector Machines (SVM) generally performed better than Random Forest (RF) and Extra Trees (ET), especially for the narrative and descriptive classes.
  • Class Imbalance Impact: The class imbalance (narrative being ~70% of the data) significantly impacted the performance of CML classifiers, leading to poor results for the minority informative class, even with class-balancing techniques.
  • Annotator Agreement: A Cohen's kappa score of 0.78 indicated good agreement among annotators.

Limitations and Future Work

  • Limited Labeled Data: The authors acknowledge that the relatively small size of the annotated dataset might limit the full potential of transformer-based models. They suggest that with more labeled data, transformer models could show even greater improvement.
  • Low-Resource Challenge: The scarcity of labeled data is a persistent challenge for low-resource languages like Bengali, making it difficult to train complex models from scratch.
  • Future Work: The authors plan to expand the size of the corpus and demonstrate the utility of this annotated data for downstream tasks such as automated essay scoring and sentiment analysis in Bengali.

Summary updated

BibTeX
@inproceedings{sazzed2022annotated,
  author = {Salim Sazzed},
  title = {{An Annotated Dataset and Automatic Approaches for Discourse Mode Identification in Low-resource Bengali Language}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure