Abstract: We propose a supervised maximum entropy ranking approach to pronoun resolution as an alternative to commonly used classification-based approaches. Classification approaches consider only one or two candidate antecedents for a pronoun at a time, whereas ranking allows all candidates to be evaluated together. We argue that this provides a more natural fit for the task than classification and show that it delivers significant performance improvements on the ACE datasets. In particular, our ranker obtains an error reduction of 9.7% over the best classification approach, the twin-candidate model. Furthermore, we show that the ranker offers some computational advantage over the twin-candidate classifier, since it easily allows the inclusion of more candidate antecedents during training. This approach leads to a further error reduction of 5.4% (a total reduction of 14.6% over the twin-candidate model).
Introduction
Pronoun resolution—the process of identifying which noun phrase (the antecedent) a pronoun refers to—is a fundamental component of natural language understanding. Traditionally, this problem has been framed as a binary classification task. In a standard single-candidate classifier (SCC), the system evaluates pronoun-candidate pairs in isolation to determine if they corefer. While more advanced "twin-candidate" models (TCC) attempt to compare pairs of candidates, they remain computationally expensive and fail to capture the global context of the resolution process.
We argue that pronoun resolution is inherently a ranking problem. Because pronouns derive their meaning from their relationship to other entities, the task is not to decide if a candidate is "good" in a vacuum, but rather to determine which candidate is the best match among all possibilities.
Methodology
We compare three distinct Maximum Entropy (MaxEnt) modeling approaches:
- Single-Candidate Classifier (SCC): A baseline that evaluates candidates individually and selects the first one exceeding a 50% probability threshold.
- Twin-Candidate Classifier (TCC): A pairwise baseline that conducts a round-robin tournament among candidates, selecting the one with the most wins.
- The Ranker (Proposed): This model evaluates the entire set of candidates for a specific pronoun simultaneously. It computes a conditional probability for each candidate relative to the whole set, selecting the candidate with the highest probability.
Feature Engineering
The models utilize a comprehensive set of features categorized into pronoun-specific, antecedent-specific, and relational attributes. These features capture linguistic properties such as gender/number agreement, grammatical roles, and the salience of the antecedent.
Experimental Setup
The study utilizes the ACE (Automatic Content Extraction) Phase 2 corpus, spanning three genres: Newspaper (NPAPER), Newswire (NWIRE), and Broadcast News (BNEWS). To isolate the performance of the resolution logic, we focus on "true" ACE mentions (gold-standard entities), ensuring that errors in mention detection do not confound the results of the resolution models.
Results and Discussion
The ranking approach consistently outperformed both classification baselines across all datasets. Using the same context window as the baselines, the ranker achieved a 9.7% error reduction over the twin-candidate model. For instance, on the NPAPER dataset, the ranker reached 76.4% accuracy, compared to 70.7% for the SCC baseline.
| System | BNEWS | NPAPER | NWIRE |
|---|---|---|---|
| Single-Candidate (SCC) | 62.2% | 70.7% | 68.3% |
| Twin-Candidate (TCC) | 68.6% | 74.6% | 71.1% |
| Ranker (RK) | 72.9% | 76.4% | 72.4% |
Scaling and Computational Efficiency
A significant advantage of the ranking approach is its computational efficiency. Unlike pairwise models that scale cubically with the number of mentions, the ranker scales linearly. This efficiency allowed us to expand the candidate search window from 4 sentences to 10 sentences ($w=10$), leading to a total error reduction of 14.6% over the twin-candidate model.
Conclusion
Shifting the paradigm of pronoun resolution from classification to ranking provides a more linguistically sound and computationally efficient framework. By evaluating all candidates globally, the ranker achieves superior accuracy and scales effectively to larger context windows. Future research will explore the integration of complex syntactic features and the performance of the ranker in end-to-end systems involving automated mention detection.
Summary updated