A High-Performance Coreference Resolution System using a Constraint-based Multi-Agent Strategy

GuoDong Zhou, Jian SuCOLING 2004

TL;DR

This paper introduces a novel multi-agent system for coreference resolution that significantly improves precision by intelligently filtering out indirect and less informative antecedents based on anaphor type. The system achieves state-of-the-art performance on MUC-6 and MUC-7 benchmarks, demonstrating the effectiveness of its constraint-based, agent-driven approach.

What Problem Does This Paper Address?

Coreference resolution, the task of identifying expressions in text that refer to the same real-world entity, is fundamental to natural language understanding. It's crucial for tasks like information extraction, machine translation, and question answering. While many systems exist, a persistent challenge is accurately linking anaphors (like pronouns or noun phrases) to their correct antecedents, especially in complex texts where the most recent mention isn't always the most direct or informative. Existing approaches often struggle with indirect coreference chains, where an anaphor is linked to an antecedent through intermediate mentions. This paper aims to tackle this by developing a system that not only identifies potential antecedents but also strategically filters them based on their directness and informativeness relative to the anaphor.

Methods

The proposed system employs a constraint-based multi-agent strategy. It breaks down the coreference resolution process into several stages:

  1. Preprocessing: Standard NLP techniques like tokenization, sentence segmentation, named entity recognition (NER), part-of-speech (POS) tagging, and noun phrase (NP) chunking are used to identify referring expressions. The NER and NP chunking components leverage HMM-based engines.
  2. Common Constraint Agent: This agent applies general knowledge to filter out obviously invalid antecedent candidates. This includes checking for morphological agreements (gender and number) and semantic consistency (using WordNet for semantic checks).
  3. Special Constraint Agents: For each identified coreference type (e.g., name alias, apposition, pronominal, definite noun phrase), a specialized agent is deployed. These agents use heuristic rules derived from the training data to filter out "less informative" antecedent candidates. A key insight here is that the most recent antecedent in a chain might be indirectly linked and thus less informative for direct resolution. These agents aim to find the most direct and informative antecedent.
  4. Common Preference Agent: After filtering, a simple preference agent selects the most likely antecedent from the remaining candidates, primarily based on the proximity principle (i.e., preferring antecedents that appear closer to the anaphor).

The system's novelty lies in its structured, multi-agent approach that differentiates coreference types and uses specialized agents to filter antecedents based on their directness and informativeness, rather than solely relying on proximity or general constraints.

Datasets

The system was trained and evaluated on datasets from the Message Understanding Conferences (MUC):

  • MUC-6: 30 dry-run documents for training and 30 standard test documents for evaluation.
  • MUC-7: 30 annotated training documents and 20 standard test documents for evaluation.

The paper notes the size of these datasets in terms of word count (approx. 12,400 words for MUC-6 training and 19,000 for MUC-7 training).

Key Insights and Ideas

  • Directness over Recency: The most significant insight is that the most recent antecedent in a coreference chain is not always the most informative for direct resolution. The system's success stems from its ability to identify and filter out these less direct antecedents.
  • Anaphor-Type Specific Filtering: Differentiating coreference types based on the anaphor (e.g., pronoun vs. definite noun phrase) allows for the application of specialized constraints, leading to more accurate filtering.
  • Multi-Agent Architecture: The modular design with specialized agents (common constraint, special constraint, preference) allows for a systematic and robust approach to coreference resolution, handling different aspects of the problem independently.
  • Balancing Precision and Recall: The system prioritizes precision by filtering out ambiguous or indirect antecedents, while still maintaining competitive recall rates.

Key Results

The system achieved strong performance on the MUC benchmarks:

  • MUC-6: Precision/Recall/F-measure of 84.7% / 65.8% / 73.9
  • MUC-7: Precision/Recall/F-measure of 82.8% / 55.7% / 66.5

These results represent a significant improvement, particularly in precision, with the paper claiming about an 8% increase over the best-reported systems while maintaining recall. Table 2 shows direct comparisons, where the proposed system outperforms Ng and Cardie's (2002) C4.5 and RIPPER systems in F-measure by 3.1-4.8 percentage points, largely due to precision gains.

Limitations and Future Work

The authors acknowledge several limitations and suggest future directions:

  • Predicate Nominal Coreference: The agent for predicate nominal coreference was still under construction at the time of writing, indicating a potential area for improvement.
  • Type Coercion/Metonymy: These phenomena, which involve semantic shifts (e.g., "The White House announced..." referring to the administration), are identified as a general problem that accounts for a significant portion of missing recall.
  • Cataphora: The current mechanism does not handle cataphora (where an anaphor precedes its antecedent).
  • New Constraints and Coreference Types: Future work could involve exploring new constraints to further improve precision and identifying new coreference types to increase recall.

Summary updated

BibTeX
@inproceedings{zhou2004high,
  author = {GuoDong Zhou and Jian Su},
  title = {{A High-Performance Coreference Resolution System using a Constraint-based Multi-Agent Strategy}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2004},
}
Figure