Description of the UPENN CAMP System as Used for Coreference

Breck Baldwin, Thomas S. Morton, Amit Bagga, Jason Baldridge, Raman Chandraseker, Alexis Dimitriadis, Kieran Snyder, Magdalena WolskaMUC 1998 · OpenReview

Abstract: Breck Baldwin, Tom Morton, Amit Bagga, Jason Baldridge, Raman Chandraseker, Alexis Dimitriadis, Kieran Snyder, Magdalena Wolska. Seventh Message Understanding Conference (MUC-7): Proceedings of a Conference Held in Fairfax, Virginia, April 29 - May 1, 1998.

1998.

Introduction

Coreference resolution—the task of identifying all mentions that refer to the same real-world entity—is a cornerstone of Information Extraction (IE). This paper details the University of Pennsylvania’s CAMP system, a modular NLP pipeline developed for the MUC-7 conference. Beyond the system itself, the authors introduce the B-CUBED algorithm, a novel scoring metric designed to address significant limitations in existing evaluation frameworks that often overestimate system performance.

The CAMP System Architecture

The CAMP system utilizes a "precision-first" modular architecture, breaking coreference down into specialized sub-tasks:

  • Genre-Specific Rules: Handling idiosyncratic text structures like news headers.
  • Noun Coreference (La Hack 3): Identifying named entities using tools like IBM’s "Textract."
  • CogNIAC: A high-precision resolution engine that uses a salience model to track entities currently "in focus" to resolve proper nouns, common nouns, and pronouns.
  • Syntactic Patterns: Identifying appositives (e.g., "John, the baker") to establish links between entities.

The B-CUBED Scoring Algorithm

A primary contribution of this work is the critique of the standard MUC scoring metric (Vilain et al.). The authors identify two critical flaws: the Singleton Problem, where the metric fails to reward the correct identification of single-mention entities, and the "All Errors are Equal" Fallacy, which ignores the disproportionate impact of merging large, distinct entity chains.

Set-based visualization of coreference partitions used in the B-CUBED algorithm to define precision and recall.

B-CUBED addresses these issues by calculating precision and recall for each individual entity. By comparing system-generated chains against a gold standard on an entity-by-entity basis, the metric provides a more intuitive reflection of system utility. This approach allows for task-dependent weighting; for example, weighting by chain size for Information Retrieval tasks.

Evaluation and Results

The system was evaluated on MUC-6 and MUC-7 datasets. The transition from Wall Street Journal (MUC-6) to New York Times (MUC-7) data highlighted the system's reliance on honorifics for gender and entity cues, which were less frequent in the latter.

Comparison of MUC and B-CUBED scoring algorithms across two examples, demonstrating the more conservative nature of B-CUBED.

As shown in the table above, the B-CUBED metric often yields lower, more realistic performance scores than the MUC algorithm. In Example 2, while the MUC algorithm reports 90% precision, B-CUBED drops to 58%, reflecting the severe impact of incorrect entity merges. Overall, the CAMP system achieved 81% precision under the MUC metric but 78% under B-CUBED. Specific modules showed varying success: the "Quoted Speech" module reached 86.8% precision, while pronoun resolution via CogNIAC struggled at 70% precision and 4.5% recall.

Conclusion

The UPENN CAMP system demonstrates the effectiveness of a modular, high-precision approach to coreference. However, the introduction of B-CUBED underscores that the choice of evaluation metric profoundly influences research direction. The authors conclude that future work must move beyond simple left-to-right processing toward entity merging models and address the complexities of common noun anaphora to overcome the "precision/recall wall."

Summary updated

BibTeX
@inproceedings{baldwin1998description,
  author = {Breck Baldwin and Thomas S. Morton and Amit Bagga and Jason Baldridge and Raman Chandraseker and Alexis Dimitriadis and Kieran Snyder and Magdalena Wolska},
  title = {{Description of the UPENN CAMP System as Used for Coreference}},
  booktitle = {MUC},
  year = {1998},
  url = {https://openreview.net/forum?id=sYm4DqM2mfK},
}
Figure