Probabilistic Coreference in Information Extraction

Andrew KehlerEMNLP 1997

Abstract

Traditional Information Extraction (IE) systems typically produce a single deterministic output for coreference resolution—the process of determining which mentions in a text refer to the same real-world entity. However, this "best guess" approach discards valuable uncertainty information, which can lead to cascading errors in downstream data fusion systems. This paper proposes a framework for generating probability distributions over possible coreference configurations. By employing Maximum Entropy models for pairwise coreference and Dempster’s Rule of Combination for global assembly, the system provides a nuanced representation of ambiguity. Evaluation on a real-world corpus demonstrates that an evidential reasoning approach significantly improves the quality of the extracted information compared to deterministic and greedy baselines.

The Architecture of Information Extraction

In modern data processing, Information Extraction serves as a bridge between unstructured natural language and structured databases. Systems like SRI’s FASTUS identify entities and events, organizing them into "templates"—structured records of attributes.

System architecture showing the role of template merging in the IE pipeline.

As shown in the architecture above, the process involves pattern recognition followed by template merging. The final output is then passed to downstream processing systems, which may aggregate data from multiple disparate sources. The critical failure point in many existing systems is the "template merging" phase, where a deterministic decision is often made without regard for the system's underlying confidence level.

The Coreference Resolution Problem

Coreference resolution is the task of deciding whether two templates represent the same entity. Consider the following extracted templates:

Example templates extracted from text, illustrating potential coreference ambiguities.

In this example, Template A and Template B both describe a "Rail Depot" numbered 1. Template C describes an "Ammunition Depot," while Template D is a generic "Depot." A deterministic system might merge A and B with 100% certainty, but the relationship between D and the others is more ambiguous. If the system is only 51% sure that D refers to the same facility as A, a deterministic output of "Merged" or "Not Merged" loses that 49% of doubt. When this data is fused with other intelligence (e.g., a satellite photo showing only one depot), the lack of probabilistic weighting can lead to incorrect conclusions.

Methodology: From Pairs to Global Distributions

The proposed framework addresses this by moving from binary decisions to probability distributions.

1. Pairwise Modeling with Maximum Entropy

The system first calculates the probability that any two templates corefer using a Maximum Entropy (MaxEnt) model. MaxEnt is chosen for its ability to incorporate diverse, overlapping features without making strong independence assumptions. Features include:

  • Linguistic Cues: Distance between mentions and the type of noun phrase (e.g., definite vs. indefinite).
  • Template Subsumption: Whether the attributes of one template are a subset of another (e.g., "Depot" is a subset of "Rail Depot").
  • Incompatibility Constraints: A priori knowledge that prevents impossible merges, such as a "Rail" type being merged with an "Ammunition" type.

2. Global Distribution Assembly

Pairwise probabilities do not naturally form a coherent global state because coreference is transitive. The paper explores two methods for assembly:

  • Merging Decision Approach: A sequential process where the probability of a final configuration is the product of individual merger decisions.
  • Evidential Reasoning (Dempster’s Rule): This approach treats each pairwise probability as an independent piece of evidence. It uses Dempster’s Rule of Combination to resolve conflicts and normalize the results into a distribution over all possible partitions of the entities.

Experimental Results

The system was tested on a development corpus of 72 messages, yielding 132 coreference sets. Performance was measured using average cross-entropy, where a lower score indicates a model that more accurately predicts the ground truth.

Approach Combined Cross-Entropy
Uniform (Baseline) 2.01
Greedy (Deterministic) 1.41
Merging Decision 1.27
Evidential (Dempster's) 1.00

The results show that the Evidential approach consistently outperforms both deterministic baselines and the sequential Merging Decision model. By considering all pairwise evidence simultaneously, the system becomes more robust to local errors.

Conclusion

The "danger of certainty" in IE systems is real: discarding ambiguity early in the pipeline prevents downstream systems from making informed decisions. By providing a mathematically sound way to quantify coreference uncertainty, this work enables more reliable data fusion. Future research will focus on scaling these probabilistic methods to larger document sets where the number of possible configurations grows exponentially.

Summary updated

BibTeX
@inproceedings{kehler1997probabilistic,
  author = {Andrew Kehler},
  title = {{Probabilistic Coreference in Information Extraction}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {1997},
}
Figure