Chinese Noun Phrase Coreference Resolution: Insights into the State of the Art

Chen Chen, Vincent NgCOLING 2012

TL;DR

This paper provides a comprehensive analysis of the state-of-the-art in Chinese noun phrase coreference resolution, a task crucial for natural language understanding. By extensively experimenting with their top-ranked system on the CoNLL-2012 shared task dataset, the authors offer valuable insights into the impact of mention detection, preprocessing, and algorithmic choices on performance.

What Problem Does This Paper Address?

Coreference resolution, the task of identifying noun phrases that refer to the same entity, is fundamental for deeper language understanding. While English coreference resolution has seen extensive research, Chinese coreference resolution has lagged, partly due to a lack of standardized evaluation datasets. The CoNLL-2012 shared task aimed to rectify this by providing a common dataset. This paper seeks to understand the current capabilities and limitations of Chinese coreference resolvers by analyzing a highly successful system.

Methods

The authors employ a hybrid rule-based/machine learning approach. Their system extends the rule-based multi-pass sieve approach with lexical features commonly used in machine learning methods. The coreference resolution process is divided into three steps:

  1. Mention Detection: This involves extracting candidate noun phrases (mentions) from syntactic parse trees and then pruning erroneous ones using both heuristic and learning-based methods.
  2. Sieve-Based Coreference Resolution: A series of manually designed rules (sieves) are applied sequentially to establish coreference links. The sieves are ordered by precision, with more precise rules applied first. The system includes 10 sieves, some of which are tailored for Chinese.
  3. Postprocessing: Coreference partitions are refined by removing singleton clusters and specific types of coreference links.

Datasets

The experiments are conducted using the training, development, and test sets provided for the CoNLL-2012 shared task, which are based on the OntoNotes corpus.

Key Insights and Ideas

  • Mention Detection is Crucial: The quality of extracted mentions (both recall and precision) significantly impacts overall coreference performance. Improving precision through better pruning strategies and recall through more effective extraction from parse trees are key areas for improvement.
  • Pruning vs. Extraction: While improving recall in mention detection is beneficial, the precision of mention detection appears to have a more substantial impact on the final coreference performance, especially when comparing gold mentions to system mentions.
  • Syntactic Parsing and NEs Have Limited Impact (for feature computation): Replacing gold-standard parse trees and named entity (NE) information with system-generated versions has a relatively small negative impact on coreference performance when used for feature computation. This suggests that the coreference algorithm is robust to some level of noise in these annotations.
  • Sieve Ordering Matters: The order in which sieves are applied is important for performance. Randomizing the sieve order leads to a noticeable drop in performance, indicating that the precision-based ordering is effective.
  • Learning Component is Significant: Removing the learning-based components of the hybrid system leads to a notable decrease in performance, highlighting the value of the machine learning aspect in their approach.
  • Chinese-Specific Rules are Beneficial: The paper implicitly suggests that the incorporated Chinese-specific rules contribute to the system's success.

Key Results

  • The authors' Chinese resolver ranked first in the CoNLL-2012 shared task on the Chinese test data.
  • Improving mention detection precision can yield substantial gains in coreference performance. For instance, the difference between using gold mentions and system mentions from system parses can be attributed approximately 58% to mention detector precision and 42% to recall.
  • Replacing gold parses with system parses and/or gold NEs with system NEs for feature computation resulted in only a minor drop in performance (1.0-1.5% Avg F-score when NE information was not used).
  • Randomly ordering the sieves decreased the Avg F-score by 1.2%, while removing the learning component decreased it by 0.6%.
  • The Chinese Head Match (CHM) and Exact String Match (ESM) sieves were found to contribute the most to performance.
  • When compared to a mention-pair (MP) model, the authors' resolver significantly outperformed MP models that used rules as features, which in turn outperformed those using atomic features.

Limitations and Future Work

  • Mention Detection Limitations: The paper acknowledges that the current mention pruning method still has room for improvement, with a precision of only 43.7% in one scenario.
  • CEAFe Scoring: The authors note that CEAFe scoring shows different trends than MUC and B³ due to its "somewhat counterintuitive definitions."
  • Generalizability of Conclusion 2: The finding that improving mention detection recall can improve coreference performance might not hold true for all languages, as suggested by previous work on English and Arabic. Further experiments are needed to confirm this.
  • Future Work: The paper implicitly suggests that further improvements in mention detection (both recall and precision) and a deeper understanding of how different sieves contribute could lead to even better performance. The comparison with the MP model also suggests that the way features are combined and learned is critical.

Summary updated

BibTeX
@inproceedings{chen2012chinese,
  author = {Chen Chen and Vincent Ng},
  title = {{Chinese Noun Phrase Coreference Resolution: Insights into the State of the Art}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2012},
}
Figure