Improving Coreference Resolution by Using Conversational Metadata

Xiaoqiang Luo, Radu Florian, Todd WardNAACL 2009

TL;DR

This paper shows that incorporating conversational metadata, specifically speaker and turn information, significantly improves coreference resolution accuracy. By treating this metadata as features in a statistical model, they achieve a statistically significant boost in performance, demonstrating its value beyond just the textual content.

What Problem Does This Paper Address?

Coreference resolution, the task of identifying all linguistic expressions that refer to the same entity in a document, is crucial for understanding text. While traditional approaches focus on textual cues, this paper tackles the challenge of leveraging structural information present in certain document types, particularly conversational ones. The authors hypothesize that metadata like speaker and turn information, which often aligns with discourse structure, can provide valuable context that is currently underutilized, especially in conversational settings where these cues are explicit.

Methods

The authors employ a data-driven, machine-learning-based coreference resolution system. Their core innovation lies in encoding conversational metadata as features within a conditional log-linear model. Specifically, for conversational documents, they create binary features that capture relationships between a candidate referent and the current mention based on whether they share the same speaker or turn, or are in consecutive turns. For non-conversational genres like newswire, they use heuristics to infer speaker and quotation information. They then compare a baseline system (without metadata features) against a system augmented with these metadata features.

Datasets

The study uses the 2007 ACE (Automatic Content Extraction) data. This dataset includes documents from various sources, including broadcast conversations, broadcast news, Usenet, web logs, and telephone conversations. The authors split this data into a training set of 499 documents and a test set of 100 documents. Importantly, their experiments are conducted on "gold-standard mentions," meaning mention detection is assumed to be perfect, and they focus solely on the coreference resolution aspect.

Key Insights and Ideas

  • Metadata as a Discourse Signal: The paper strongly suggests that metadata like speaker and turn information isn't just auxiliary data; it directly reflects the underlying discourse structure, which is vital for coreference resolution.
  • Pronoun Resolution Boost: The metadata features are particularly effective in linking pronouns to their antecedents, especially in conversational contexts where speaker shifts are common.
  • Beyond Textual Content: This work highlights that valuable information for NLP tasks can exist outside the raw text, in the form of structural or contextual metadata.
  • Impact on Different Metrics: While the standard ACE-value metric shows a modest improvement, a less common metric (B³) that weights all mention types equally reveals a much larger gain, suggesting metadata's impact is more profound than initially apparent.

Key Results

The coreference resolution system augmented with speaker and turn metadata achieved an overall 0.7 point improvement in ACE-value compared to the baseline system. This difference was found to be statistically significant (p < 0.002) using a paired Wilcoxon rank-sum test. When evaluating using the B³ metric, which gives equal weight to all mention types, the improvement was even more substantial, rising from 73.8% to 76.4% (a 2.6 point improvement), which is nearly four times larger than the ACE-value change. For telephone conversations specifically, the ACE-value improved by 4.6 points.

Limitations and Future Work

The authors acknowledge that for non-conversational genres, inferring speaker and quotation information relies on heuristics, which may not always be accurate. They also note that the ACE evaluation metric (ACE-value) heavily discounts pronominal mentions and entities, potentially masking the full impact of metadata. Future work could explore weighing all mention types equally to better capture the benefits of metadata. Additionally, the paper suggests that further investigation into other types of metadata and their integration into coreference resolution systems could yield even greater improvements.

Summary updated

BibTeX
@inproceedings{luo2009improving,
  author = {Xiaoqiang Luo and Radu Florian and Todd Ward},
  title = {{Improving Coreference Resolution by Using Conversational Metadata}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2009},
}
Figure