Quality over Quantity: Rethinking Data and Model Assumptions for Bengali Coreference Resolution

Zenith Biswas, Andrew Thomas DyerarXiv 2026 · arXiv

In the world of Natural Language Processing (NLP), the prevailing wisdom is often "more data is better." However, for the 250 million speakers of Bengali, high-quality annotated data is a rare luxury. In their paper, "Quality over Quantity: Rethinking Data and Model Assumptions for Bengali Coreference Resolution," Zenith Biswas and Andrew Thomas Dyer challenge standard low-resource NLP assumptions, demonstrating that when it comes to complex tasks like coreference resolution, how you build your dataset matters far more than how big it is.

TL;DR

This paper shows that a tiny set of 41 high-quality "gold" documents outperforms much larger datasets of mixed quality for Bengali coreference resolution. The authors also identify a critical "embedding collapse" issue in several popular multilingual models and find that standard data augmentation techniques like back-translation can actually degrade performance.

What Problem Does This Paper Address?

Coreference resolution—the task of identifying all expressions in a text that refer to the same entity—is a fundamental building block of natural language understanding. While English models benefit from massive datasets like OntoNotes, Bengali is severely under-resourced.

Beyond data scarcity, Bengali presents unique linguistic hurdles:

  • Gender-neutral pronouns: The same pronoun can refer to any gender.
  • Complex honorifics: Pronouns like tini (formal "he/she") are shared across all respected referents in a passage, leading models to "over-link" different people into a single entity cluster.

The authors set out to see if common "low-resource" strategies—like data augmentation, cross-lingual transfer, and using massive multilingual models—actually work for this specific, span-sensitive task.

Methods

The researchers evaluated 10 pre-trained transformer encoders. Because they were working with as few as 41 training documents, they used a frozen encoder architecture with a trainable three-layer MLP classifier. This prevents the instability and "forgetting" that often happens when full fine-tuning is attempted on tiny datasets.

They tested these models across nine configurations, including:

  • Zero-shot: Testing models with no task-specific training.
  • Supervised Fine-tuning: Using varying amounts of gold (human-annotated) and silver (machine-translated/projected) data.
  • Data Augmentation: Using back-translation and Masked Language Model (MLM) paraphrasing.
  • Cross-lingual Transfer: Training on Hindi (linguistically related) or English (linguistically distant) data.

Datasets

The study utilized four key datasets to isolate the effects of data quality vs. quantity:

Dataset statistics showing the small size of Gold data vs. Silver alternatives
  • BenCoref: The only gold-standard Bengali coreference dataset (122 documents).
  • TransMuCoRes-BN: A "silver" dataset created by machine-translating English text into Bengali and automatically projecting the annotations.
  • TransMuCoRes-HI: A similar silver dataset for Hindi, used for cross-lingual transfer.
  • OntoNotes 5.0 (subset): A subsampled English dataset used as a distant transfer source.

Key Insights and Ideas

  • The "Embedding Collapse" Diagnostic: The authors discovered that several popular models (like XLM-RoBERTa and MuRIL-Base) fail in Bengali because their embedding spaces "collapse." In these models, the mathematical similarity between two mentions of the same person is nearly identical to the similarity between two mentions of different people. Embedding discriminability across all ten candidate models
  • Augmentation Failure: Back-translation, usually a "free win" in NLP, proved harmful. Because coreference depends on exact word spans, the sentence restructuring caused by back-translation often shifted labels onto the wrong words, effectively poisoning the training data.
  • Structural Signal vs. Language Knowledge: Surprisingly, BERT-Base-Uncased—a model with zero Bengali in its pre-training—achieved the highest zero-shot score. This suggests that coreference resolution may rely heavily on structural patterns in contextual embeddings that transcend specific languages.
Figure 1: Embedding discriminability across all ten can- didate models. Green bars show mean within-cluster (coreferent) similarity; red bars show mean between- cluster (non-coreferent) similarity. The gap (∆) indi- cates discriminability. Models in the shaded region have gaps below 0.01 and are unsuitable for threshold-based coreference clustering.

Key Results

  • Quality Wins: Training on just 41 gold documents (64.36% F1) significantly outperformed training on a mixed set of 141 documents (63.59% F1) that included 100 silver documents. The noise introduced by automatic projection outweighed the benefit of a 3x larger dataset.
  • Top Performer: MuRIL-Large achieved the best overall result at 66.73% CoNLL F1 when trained on gold data.
  • Transfer Inconsistency: Transferring from English reduced performance, and Hindi transfer only provided a meaningful boost for the largest model (MuRIL-Large), suggesting that linguistic similarity alone isn't a silver bullet for transfer learning.
CoNLL F1 (%) across all models and training configurations

Limitations and Future Work

The authors acknowledge that their study is limited to a single language and a specific "frozen encoder" architecture. Furthermore, the BenCoref test set is relatively small (71 documents), which limits the statistical power to detect very subtle effects.

Future research could explore "span-aware" augmentation methods that preserve alignment during translation, or investigate whether these findings hold true for other Indo-Aryan languages facing similar resource constraints.

Summary updated

Abstract

Multilingual transformer models are often assumed to generalise across languages, but their effectiveness for coreference resolution in low-resource settings remains unverified. We investigate this gap in Bengali, a language with over 250 million speakers but limited annotated coreference resources. We evaluate transformer-based pre-trained encoders under multiple settings, including zero-shot, fine-tuning, data augmentation, and cross-lingual transfer. Our results challenge several common assumptions about model generalisation and data efficiency in low-resource NLP. We demonstrate that a small amount of high-quality annotated data significantly outperforms larger mixed-quality datasets (p < 0.01). We find that data augmentation, particularly back-translation, does not improve performance and can degrade results in coreference resolution. Cross-lingual transfer is also inconsistent: transfer from a distant language like English reduces performance, and transfer from the related language Hindi provides no consistent benefit across models. Several widely used multilingual models exhibit collapsed embedding spaces, rendering them unsuitable for similarity-based clustering tasks. These findings highlight the importance of annotation quality over data quantity and provide empirical guidance for developing coreference resolution systems in low-resource languages.
BibTeX
@misc{biswas2026quality,
  author = {Zenith Biswas and Andrew Thomas Dyer},
  title = {{Quality over Quantity: Rethinking Data and Model Assumptions for Bengali Coreference Resolution}},
  year = {2026},
  eprint = {2610.35114},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.35114},
}
Figure