Using the Web in Machine Learning for Other-Anaphora Resolution

Natalia N. Modjeska, Katja Markert, Malvina NissimEMNLP 2003 · OpenReview

Abstract

We present a machine learning framework for resolving other-anaphora. Besides morpho-syntactic, recency, and semantic features based on existing lexical knowledge resources, our algorithm obtains additional semantic knowledge from the Web. We search the Web via lexico-syntactic patterns that are specific to other-anaphors. Incorporating this innovative feature leads to an 11.4 percentage point improvement in the classifier's F-measure (25% improvement relative to results without this feature).

Introduction

In natural language processing, most work on anaphora resolution focuses on identity—determining that a pronoun like "he" refers to a previously mentioned entity like "John." However, a more complex challenge exists in resolving other-anaphora. These are referential noun phrases (NPs) modified by "other" or "another," such as "eight other Soviet cities" or "another risk factor."

Resolving these requires understanding set-exclusion relationships. For example, in the sentence, "An exhibition opened in Moscow and will travel to eight other Soviet cities," the system must identify "Moscow" as the antecedent by recognizing it as a member of the set "Soviet cities." Traditional lexical resources like WordNet often fail to capture these context-specific or informal relationships, such as recognizing "age" as a "risk factor" in a financial context. This "knowledge gap" is the primary obstacle this research addresses.

Methodology: The Web as a Live Ontology

The authors propose a machine learning framework using a Naive Bayes (NB) classifier to identify the most likely antecedent from a set of candidates. The system utilizes two primary feature sets:

  1. F1 (Standard Features): This baseline set includes morpho-syntactic data (gender, number), recency (sentence distance), and semantic roles. It also leverages WordNet to identify hypernymy (is-a) and meronymy (part-of) relations.
  2. Web-Based Features: To overcome the limitations of static ontologies, the authors treat the Web as a massive, real-time corpus. They employ specific lexico-syntactic patterns to validate semantic relationships.

Lexico-Syntactic Patterns

The core innovation involves searching the Web for patterns that explicitly link a candidate antecedent with the anaphoric phrase. For instance, the pattern "[Candidate] and other [Anaphor]" provides strong evidence of a set-membership relationship.

Lexico-syntactic patterns and their instantiations for common nouns and proper names.

The system uses the Google API to retrieve hit counts for these patterns. To rank candidates, a score is calculated based on Mutual Information (MI), which compares the frequency of the pattern against the individual frequencies of the candidate and the anaphor. This ensures the relationship is statistically significant and not just a result of common words appearing together. The candidate with the highest MI score is flagged with a "webfirst" feature, providing a powerful signal to the classifier.

Experimental Setup

The study utilized a dataset of 500 other-anaphors extracted from the Wall Street Journal corpus (Penn Treebank, release 2). Correct antecedents were manually annotated, resulting in 3,084 antecedent-anaphor pairs (500 positive, 2,584 negative). The classifier was evaluated using 10-fold cross-validation.

Results

The integration of Web-based features significantly enhanced the resolution performance. While standard features (F1) provided a solid foundation, the addition of Web-mined knowledge led to a substantial jump in precision, recall, and F-measure.

Performance comparison of the baseline, standard features (F1), and the combined feature set (F1+Web).

As shown in the results, the addition of Web-based features provided an 11.4 percentage point improvement in F-measure, representing a 25% relative gain over the standard feature set. This demonstrates that the Web can effectively bridge the knowledge gap where traditional ontologies fall short.

Discussion and Limitations

Despite the success, several challenges remain:

  • Named Entity Recognition (NER): The system is sensitive to NER errors. If a location like "Moscow" is misclassified as a person, the Web search patterns will likely fail to find relevant matches.
  • Pronouns: Web-search patterns are currently ineffective when the antecedent is a pronoun (e.g., "It and other cities"), as pronouns lack the lexical content needed for meaningful pattern matching. This accounted for approximately 7.4% of errors.
  • Situational Knowledge: Some relationships are highly specific to a niche context or a single speaker, making them unlikely to appear in general Web patterns.
  • Classifier Architecture: The current model treats pairs independently. Future work could explore "competition learning" or incremental processing to better simulate human cognitive processes in anaphora resolution.

Conclusion

By leveraging the Web as a "live ontology," this framework provides a scalable solution for resolving complex semantic relationships in other-anaphora. The significant performance gains highlight the potential of using large-scale Web data to supplement traditional lexical resources in machine learning tasks.

Summary updated

Abstract

We present a machine learning framework for resolving other-anaphora. Besides morpho-syntactic, recency, and semantic features based on existing lexical knowledge resources, our algorithm obtains additional semantic knowledge from the Web. We search the Web via lexico-syntactic patterns that are specific to other-anaphors. Incorporating this innovative feature leads to an 11.4 percentage point improvement in the classifier's F-measure (25% improvement relative to results without this feature).
BibTeX
@inproceedings{modjeska2003using,
  author = {Natalia N. Modjeska and Katja Markert and Malvina Nissim},
  title = {{Using the Web in Machine Learning for Other-Anaphora Resolution}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2003},
  url = {https://openreview.net/forum?id=B1-FdzfO-H},
}
Figure