Ensemble Transfer Learning for Multilingual Coreference Resolution

Tuan Lai, Heng Ji

TL;DR

This paper tackles the challenge of entity coreference resolution for languages with limited annotated data by proposing an ensemble transfer learning framework. They combine multiple transfer learning techniques and introduce a novel, low-cost method that leverages Wikipedia hyperlinks to create distantly-supervised datasets, achieving state-of-the-art results on several non-English languages.

What Problem Does This Paper Address?

Entity coreference resolution, the task of identifying mentions in text that refer to the same real-world entity, is crucial for many NLP applications like information extraction and question answering. While extensively studied for English, it remains a significant challenge for other languages due to the scarcity of labeled training data. Existing transfer learning methods often address this by using a single technique, but this paper argues that combining multiple techniques can be more effective.

Methods

The authors propose an ensemble-based framework for multilingual coreference resolution. The core idea is to train multiple coreference resolution models using different transfer learning (TL) techniques and then combine their predictions during inference.

The TL techniques employed are:

  • Continued Training: A model pre-trained on a source dataset (English OntoNotes) is fine-tuned on a smaller target dataset.
  • Joint Training: A model is trained on a combined dataset of both source and target data.
  • Wikipedia Pre-Training (Novel Method): This is a key contribution. The authors leverage the observation that anchor texts pointing to the same Wikipedia page are likely coreferential. They build a large, distantly-supervised dataset for the target language by extracting these links from Wikipedia. A model is then pre-trained on this Wikipedia-derived dataset before being fine-tuned on the actual target dataset.

For inference, the framework uses a simple unweighted averaging of the prediction scores from the individual models. The underlying coreference resolution model architecture is a standard span-based end-to-end model, similar to existing work.

Datasets

The paper uses the following datasets:

  • Source Dataset:
    • OntoNotes English (for pre-training cross-lingual models)
  • Target Datasets:
    • OntoNotes Arabic
    • SemEval Dutch
    • SemEval Spanish

Additionally, they construct a Wikipedia-based dataset for each target language (Arabic, Dutch, Spanish) using hyperlinks as a source of distant supervision.

Key Insights and Ideas

  • Ensembling is Key: Combining multiple transfer learning techniques, even simple ones, significantly boosts performance compared to using any single method. This highlights the complementary nature of different TL approaches.
  • Wikipedia as a Data Goldmine: The novel Wikipedia-based pre-training method is highly effective, especially in low-resource settings, and crucially, it does not require any labeled source data. This democratizes access to effective TL for languages with minimal resources.
  • Diversity Matters: Combining models trained with different TL methods yields better results than combining models trained with the same baseline approach but different TL strategies. This underscores the importance of model diversity in ensembles.
  • Baseline Performance: Even the simple baseline approach (training directly on the target dataset) surprisingly outperformed previous state-of-the-art (SOTA) on SemEval Spanish, suggesting potential undertraining in prior work.

Key Results

  • The proposed ensemble methods consistently outperform the baseline approach by up to 7.68% absolute gain in F1 score.
  • The best ensembles achieve new state-of-the-art (SOTA) results for three languages: Arabic, Dutch, and Spanish.
  • For Arabic, the best ensemble achieved an F1 score of 66.72%, outperforming prior work by 2.17%.
  • For Dutch, the best ensemble achieved 60.49%, outperforming prior work by 5.09%.
  • For Spanish, the best ensemble achieved 75.62%, outperforming prior work by 24.32%.
  • In extremely low-resource settings (10 documents for Arabic training, no source dataset), the ensemble approach improved performance by 5.18% over the baseline.

Limitations and Future Work

  • Social Biases: The use of pre-trained multilingual models like XLM-R and GigaBERT means the coreference resolution models may inherit and amplify social biases present in the web data they were trained on. The authors plan to work on mitigating these biases.
  • Performance Ceiling: Despite achieving SOTA, the performance is not perfect. For OntoNotes Arabic, the best system achieved an F1 score of 66.72%, which might not be sufficient for some critical downstream tasks.
  • Rare Languages: While Wikipedia is available in many languages, very rare languages might have insufficient Wikipedia pages, limiting the effectiveness of the Wikipedia-based method.
  • Future Work: The authors suggest exploring more context-dependent ensemble methods (beyond simple averaging) and investigating model compression techniques to reduce computational complexity.

Summary updated

Abstract

Entity coreference resolution is an important research problem with many applications, including information extraction and question answering. Coreference resolution for English has been studied extensively. However, there is relatively little work for other languages. A problem that frequently occurs when working with a non-English language is the scarcity of annotated training data. To overcome this challenge, we design a simple but effective ensemble-based framework that combines various transfer learning (TL) techniques. We first train several models using different TL methods. Then, during inference, we compute the unweighted average scores of the models' predictions to extract the final set of predicted clusters. Furthermore, we also propose a low-cost TL method that bootstraps coreference resolution models by utilizing Wikipedia anchor texts. Leveraging the idea that the coreferential links naturally exist between anchor texts pointing to the same article, our method builds a sizeable distantly-supervised dataset for the target language that consists of tens of thousands of documents. We can pre-train a model on the pseudo-labeled dataset before finetuning it on the final target dataset. Experimental results on two benchmark datasets, OntoNotes and SemEval, confirm the effectiveness of our methods. Our best ensembles consistently outperform the baseline approach of simple training by up to 7.68% in the F1 score. These ensembles also achieve new state-of-the-art results for three languages: Arabic, Dutch, and Spanish.
BibTeX
@inproceedings{lai2023ensemble,
  author = {Tuan Lai and Heng Ji},
  title = {{Ensemble Transfer Learning for Multilingual Coreference Resolution}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure