TL;DR
This paper tackles the challenge of entity coreference resolution for languages with limited annotated data by proposing an ensemble transfer learning framework. They combine multiple transfer learning techniques and introduce a novel, low-cost method that leverages Wikipedia hyperlinks to create distantly-supervised datasets, achieving state-of-the-art results on several non-English languages.
What Problem Does This Paper Address?
Entity coreference resolution, the task of identifying mentions in text that refer to the same real-world entity, is crucial for many NLP applications like information extraction and question answering. While extensively studied for English, it remains a significant challenge for other languages due to the scarcity of labeled training data. Existing transfer learning methods often address this by using a single technique, but this paper argues that combining multiple techniques can be more effective.
Methods
The authors propose an ensemble-based framework for multilingual coreference resolution. The core idea is to train multiple coreference resolution models using different transfer learning (TL) techniques and then combine their predictions during inference.
The TL techniques employed are:
- Continued Training: A model pre-trained on a source dataset (English OntoNotes) is fine-tuned on a smaller target dataset.
- Joint Training: A model is trained on a combined dataset of both source and target data.
- Wikipedia Pre-Training (Novel Method): This is a key contribution. The authors leverage the observation that anchor texts pointing to the same Wikipedia page are likely coreferential. They build a large, distantly-supervised dataset for the target language by extracting these links from Wikipedia. A model is then pre-trained on this Wikipedia-derived dataset before being fine-tuned on the actual target dataset.
For inference, the framework uses a simple unweighted averaging of the prediction scores from the individual models. The underlying coreference resolution model architecture is a standard span-based end-to-end model, similar to existing work.
Datasets
The paper uses the following datasets:
- Source Dataset:
- OntoNotes English (for pre-training cross-lingual models)
- Target Datasets:
- OntoNotes Arabic
- SemEval Dutch
- SemEval Spanish
Additionally, they construct a Wikipedia-based dataset for each target language (Arabic, Dutch, Spanish) using hyperlinks as a source of distant supervision.
Key Insights and Ideas
- Ensembling is Key: Combining multiple transfer learning techniques, even simple ones, significantly boosts performance compared to using any single method. This highlights the complementary nature of different TL approaches.
- Wikipedia as a Data Goldmine: The novel Wikipedia-based pre-training method is highly effective, especially in low-resource settings, and crucially, it does not require any labeled source data. This democratizes access to effective TL for languages with minimal resources.
- Diversity Matters: Combining models trained with different TL methods yields better results than combining models trained with the same baseline approach but different TL strategies. This underscores the importance of model diversity in ensembles.
- Baseline Performance: Even the simple baseline approach (training directly on the target dataset) surprisingly outperformed previous state-of-the-art (SOTA) on SemEval Spanish, suggesting potential undertraining in prior work.
Key Results
- The proposed ensemble methods consistently outperform the baseline approach by up to 7.68% absolute gain in F1 score.
- The best ensembles achieve new state-of-the-art (SOTA) results for three languages: Arabic, Dutch, and Spanish.
- For Arabic, the best ensemble achieved an F1 score of 66.72%, outperforming prior work by 2.17%.
- For Dutch, the best ensemble achieved 60.49%, outperforming prior work by 5.09%.
- For Spanish, the best ensemble achieved 75.62%, outperforming prior work by 24.32%.
- In extremely low-resource settings (10 documents for Arabic training, no source dataset), the ensemble approach improved performance by 5.18% over the baseline.
Limitations and Future Work
- Social Biases: The use of pre-trained multilingual models like XLM-R and GigaBERT means the coreference resolution models may inherit and amplify social biases present in the web data they were trained on. The authors plan to work on mitigating these biases.
- Performance Ceiling: Despite achieving SOTA, the performance is not perfect. For OntoNotes Arabic, the best system achieved an F1 score of 66.72%, which might not be sufficient for some critical downstream tasks.
- Rare Languages: While Wikipedia is available in many languages, very rare languages might have insufficient Wikipedia pages, limiting the effectiveness of the Wikipedia-based method.
- Future Work: The authors suggest exploring more context-dependent ensemble methods (beyond simple averaging) and investigating model compression techniques to reduce computational complexity.
Summary updated