A Benchmark of Rule-Based and Neural Coreference Resolution in Dutch Novels and News

Corbèn Poot, Andreas van CranenburghCOLING 2020

TL;DR

This paper benchmarks a rule-based and a neural coreference resolution system on Dutch novels and news. It reveals that while neural systems excel on news, rule-based systems perform better on literary texts, highlighting the impact of domain and data characteristics on coreference performance.

What Problem Does This Paper Address?

Coreference resolution, the task of identifying mentions in text that refer to the same real-world entity, is crucial for natural language understanding. While English has seen significant advancements with end-to-end neural models, Dutch has lagged, relying on older rule-based or machine learning approaches. This paper aims to bridge this gap by providing a direct comparison of a state-of-the-art neural system (e2e-Dutch) against a strong rule-based system (dutchcoref) on two distinct Dutch datasets: a news/Wikipedia corpus (SoNaR-1) and a corpus of contemporary novels (RiddleCoref). The goal is to understand the strengths and weaknesses of each approach across different domains and document types, shedding light on the challenges specific to Dutch coreference.

Methods

The study compares two coreference resolution systems:

  • dutchcoref: A rule-based system based on the Stanford coreference system (Lee et al., 2011, 2013). It utilizes Alpino parse trees and infers information about speakers and addressees of direct speech using heuristic rules. Some improvements were made to adapt it to the SoNaR-1 annotations.
  • e2e-Dutch: A fully end-to-end neural system, adapted from Lee et al. (2018). It uses BERT token embeddings and is trained solely on token and coreference information from CoNLL files, without external metadata like speaker information. The model incorporates character n-gram embeddings, fastText embeddings, and BERT embeddings, fed into a bidirectional LSTM. It also includes support for singletons during mention detection.

The evaluation focuses on standard metrics like CoNLL score and LEA (which gives more weight to larger entities).

Datasets

The paper uses two Dutch coreference corpora:

  • SoNaR-1: A large, genre-balanced corpus of 1 million words, predominantly news and Wikipedia text. Annotations were created semi-automatically. The authors created a 70/15/15 train/dev/test split and converted annotations to CoNLL-2012 format.
  • RiddleCoref: A corpus of contemporary Dutch novels, presented by van Cranenburgh (2019). This dataset was annotated by correcting the output of dutchcoref. It consists of novel fragments and is significantly smaller than SoNaR-1, with a different annotation scheme (manual correction of mentions, including singletons). A 70% training split was created for this dataset.

Key Insights and Ideas

  • Domain Matters: The most striking finding is the domain-specific performance difference. Neural systems shine on news/Wikipedia text, while rule-based systems are superior for literary novels. This suggests that the characteristics of literary text (e.g., longer documents, more complex narrative structures) pose unique challenges that rule-based systems, with their explicit knowledge encoding, are better equipped to handle.
  • Neural Systems' Weaknesses: The neural system (e2e-Dutch) struggles with long documents, showing a performance decrease as document length increases. This is likely due to the difficulty of maintaining long-range dependencies and the computational cost associated with processing extensive texts. Limited training data also appears to be a bottleneck for the neural model.
  • Rule-Based Systems' Sensitivity: The rule-based system (dutchcoref) is affected by annotation differences between datasets. While it performs well on RiddleCoref, its performance dips on SoNaR-1, partly due to the semi-automatic annotation scheme and the inclusion of non-referring expressions as markables.
  • Mention Detection is Key: The paper emphasizes that mention detection significantly impacts coreference resolution performance. Errors in identifying mentions, especially singletons and non-referential expressions, can cascade into coreference link errors.

Key Results

  • RiddleCoref (Novels): dutchcoref outperforms e2e-Dutch by a 6-point margin on the CoNLL score (65.91 vs. 64.81 on the dev set, and 69.86 vs. 63.55 on the test set).
  • SoNaR-1 (News/Wikipedia): e2e-Dutch significantly outperforms dutchcoref, achieving a CoNLL score of 71.53 on the dev set and 68.45 on the test set, compared to 55.45 and 55.91 for dutchcoref, respectively.
  • Document Length Impact: For e2e-Dutch on RiddleCoref, coreference scores show a negative correlation with document length, indicating poorer performance on longer texts. This effect was not observed for dutchcoref.
  • Error Analysis: e2e-Dutch makes fewer "extra" mention errors but more "missing" mention errors, particularly on names and nominals. Both systems struggle with pronoun coreference, which is identified as a general challenge in Dutch.

Limitations and Future Work

  • Dataset Comparability: The authors acknowledge that direct comparison between the two datasets is hindered by differences in document length, domain, and mention annotation schemes.
  • Neural System Training: The neural system's performance might improve with more training data, as indicated by its learning curve not reaching a plateau.
  • Rule-Based System Adaptation: dutchcoref could benefit from better adaptation to the SoNaR-1 annotation scheme.
  • Future Work: The authors propose several avenues for future research:
  • Improving the neural system by incorporating genre metadata and finetuning BERT.
  • Extending the rule-based system into a hybrid approach with supervised classifiers.
  • Harmonizing annotations across corpora through manual mention annotation to improve comparability.
  • A more systematic study of SoNaR-1 annotation issues.

Summary updated

BibTeX
@inproceedings{poot2020benchmark,
  author = {Corbèn Poot and Andreas van Cranenburgh},
  title = {{A Benchmark of Rule-Based and Neural Coreference Resolution in Dutch Novels and News}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2020},
}
Figure