Abstract: Breck Baldwin, Tom Morton, Amit Bagga, Jason Baldridge, Raman Chandraseker, Alexis Dimitriadis, Kieran Snyder, Magdalena Wolska. Seventh Message Understanding Conference (MUC-7): Proceedings of a Conference Held in Fairfax, Virginia, April 29 - May 1, 1998.
1998.
Introduction
Coreference resolution—the task of identifying all mentions that refer to the same real-world entity—is a cornerstone of Information Extraction (IE). This paper details the University of Pennsylvania’s CAMP system, a modular NLP pipeline developed for the MUC-7 conference. Beyond the system itself, the authors introduce the B-CUBED algorithm, a novel scoring metric designed to address significant limitations in existing evaluation frameworks that often overestimate system performance.
The CAMP System Architecture
The CAMP system utilizes a "precision-first" modular architecture, breaking coreference down into specialized sub-tasks:
- Genre-Specific Rules: Handling idiosyncratic text structures like news headers.
- Noun Coreference (La Hack 3): Identifying named entities using tools like IBM’s "Textract."
- CogNIAC: A high-precision resolution engine that uses a salience model to track entities currently "in focus" to resolve proper nouns, common nouns, and pronouns.
- Syntactic Patterns: Identifying appositives (e.g., "John, the baker") to establish links between entities.
The B-CUBED Scoring Algorithm
A primary contribution of this work is the critique of the standard MUC scoring metric (Vilain et al.). The authors identify two critical flaws: the Singleton Problem, where the metric fails to reward the correct identification of single-mention entities, and the "All Errors are Equal" Fallacy, which ignores the disproportionate impact of merging large, distinct entity chains.
B-CUBED addresses these issues by calculating precision and recall for each individual entity. By comparing system-generated chains against a gold standard on an entity-by-entity basis, the metric provides a more intuitive reflection of system utility. This approach allows for task-dependent weighting; for example, weighting by chain size for Information Retrieval tasks.
Evaluation and Results
The system was evaluated on MUC-6 and MUC-7 datasets. The transition from Wall Street Journal (MUC-6) to New York Times (MUC-7) data highlighted the system's reliance on honorifics for gender and entity cues, which were less frequent in the latter.
As shown in the table above, the B-CUBED metric often yields lower, more realistic performance scores than the MUC algorithm. In Example 2, while the MUC algorithm reports 90% precision, B-CUBED drops to 58%, reflecting the severe impact of incorrect entity merges. Overall, the CAMP system achieved 81% precision under the MUC metric but 78% under B-CUBED. Specific modules showed varying success: the "Quoted Speech" module reached 86.8% precision, while pronoun resolution via CogNIAC struggled at 70% precision and 4.5% recall.
Conclusion
The UPENN CAMP system demonstrates the effectiveness of a modular, high-precision approach to coreference. However, the introduction of B-CUBED underscores that the choice of evaluation metric profoundly influences research direction. The authors conclude that future work must move beyond simple left-to-right processing toward entity merging models and address the complexities of common noun anaphora to overcome the "precision/recall wall."
Summary updated