BenCoref: A Multi-Domain Dataset of Nominal Phrases and Pronominal Reference Annotations

Shadman Rohan, Mojammel Hossain, Mohammad Mamun Or Rashid, Nabeel MohammedACL 2023

TL;DR

This paper introduces BenCoref, the first multi-domain dataset for coreference resolution in Bengali, a low-resource language. This resource is crucial for advancing NLP tasks in Bengali, as existing datasets are scarce and existing models struggle with cross-lingual transfer from English.

What Problem Does This Paper Address?

Coreference resolution, the task of identifying all mentions referring to the same entity in a text, is a fundamental NLP problem. While well-studied for resource-rich languages like English, it remains significantly underexplored for Bengali. This is primarily due to the lack of publicly available annotated datasets. Bengali, with its rich morphology and linguistic nuances like zero anaphora, presents unique challenges that cannot be directly addressed by models trained on English data. The authors aim to bridge this gap by creating a comprehensive Bengali coreference dataset and demonstrating its utility.

Methods

The authors developed a novel dataset, BenCoref, by annotating Bengali texts from four diverse domains: biography, descriptive texts (similar to Wikipedia), stories, and novels. The annotation process involved adapting the OntoNotes guidelines, a standard in the field, to suit Bengali's linguistic characteristics. They used the WebAnno tool for annotation and implemented a rigorous process to ensure consistency, including inter-annotator agreement checks. For experimentation, they employed an end-to-end neural network model, leveraging pre-trained embeddings (FastText, GloVe) and transformer models (BERT), and adapted a span-ranking head for coreference resolution.

Datasets

The primary contribution of this paper is the introduction of BenCoref, a new dataset for Bengali coreference resolution. It comprises 48,569 tokens with 5,200 mention annotations forming 502 mention clusters across four domains: biography, descriptive, story, and novel. The dataset is available for public use. The authors also reference existing large-scale coreference datasets like ACE and OntoNotes, which have been instrumental in advancing research in English and other languages.

Key Insights and Ideas

  • Bengali's Linguistic Uniqueness: The paper highlights specific linguistic features of Bengali that pose challenges for coreference resolution, such as zero anaphora (implicit subjects/objects), non-anaphoric pronouns, and the impact of case marking on disambiguating entity roles.
  • Low-Resource Language Challenges: The scarcity of annotated data for Bengali significantly hinders the performance of even state-of-the-art models, especially in zero-shot cross-lingual settings from English.
  • Domain-Specific Variations: Coreference phenomena exhibit variations across different domains. For instance, biographies tend to have shorter distances between mentions, potentially leading to better model performance in that domain.
  • The Need for Language-Specific Resources: The poor performance of multilingual BERT in a zero-shot cross-lingual setting underscores the critical need for dedicated, high-quality datasets for each language to achieve robust coreference resolution.

Key Results

The authors report that their end-to-end neural network model achieved reasonable performance on the BenCoref dataset. While identifying individual mentions was generally successful (as seen in Table 4), the clustering phase presented more challenges, particularly with complex sentence structures and lexical ambiguity. The model performed best on the biography domain, likely due to the shorter mention distances. The zero-shot cross-lingual experiment using multilingual BERT yielded very low F1 scores (around 5-7%), emphasizing the limitations of transferring knowledge across languages without specific training data.

Limitations and Future Work

The authors acknowledge that the BenCoref dataset is relatively small, which might limit the performance of neural models. They also note that the process of resolving disagreements during annotation was not fully documented and will be addressed in future work. A recurring problem identified was the model's difficulty in handling long coreference clusters, sometimes breaking them into multiple incorrect clusters. Future work could involve expanding the dataset, exploring projection techniques for improved cross-lingual coreference resolution, and further investigating the linguistic nuances of Bengali coreference.

Summary updated

BibTeX
@inproceedings{rohan2023bencoref,
  author = {Shadman Rohan and Mojammel Hossain and Mohammad Mamun Or Rashid and Nabeel Mohammed},
  title = {{BenCoref: A Multi-Domain Dataset of Nominal Phrases and Pronominal Reference Annotations}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure