Adapting a synonym database to specific domains

Davide Turcato, Fred Popowich, Janine Toole, Dan Fass, Devlan Nicholson, Gordon TisherACL 2000

TL;DR

This paper presents a method for tailoring general-purpose synonym databases, like WordNet, to specific technical domains. By intelligently pruning irrelevant or incorrect synonymy relations, the authors demonstrate how to significantly improve the accuracy and efficiency of domain-specific applications, such as Information Retrieval.

What Problem Does This Paper Address?

Information Retrieval (IR) systems often benefit from using synonyms to expand query terms, helping to find relevant documents that might not use the exact same wording. However, a major challenge arises when applying these systems to specialized technical domains. General-purpose synonym databases, like the widely-used WordNet, contain a vast number of synonymy relations that are valid for the language at large but can be detrimental in a specific domain. For instance, the synonymy between "snow" and "C" (for Celsius) is nonsensical in a weather report context, yet both words are common. This overgeneration of incorrect synonymy relations leads to a drop in precision, meaning the system returns many irrelevant results. The core problem this paper addresses is how to adapt these broad synonym databases to be accurate and efficient for specific technical domains by removing these problematic relations.

Methods

The authors propose an eliminative approach to adapt a general synonym database (WordNet) to a specific domain. This process involves incrementally pruning the original database. The methodology is broken down into three main steps:

  1. Manual Pruning: This initial phase focuses on the most impactful synonymy relations. It involves:
  2. Ranking Synonymy Relations: Relations are scored based on how likely they are to be used in the domain. This ranking prioritizes polysemous terms over monosemous ones and considers the frequency of terms in both a domain-specific corpus and a query inventory.
  3. Correctness Evaluation: Human evaluators manually assess the top-ranked synonymy relations, deciding whether a term truly belongs to a given synset within the specific domain. Evidence is drawn from corpus examples.
  4. Automatic Pruning: This step automates the process of identifying and ranking potentially incorrect synonymy relations. It uses corpus frequencies of words within synsets and their glosses to assign scores. The goal is to identify (term, sense) pairs that are unlikely to be correct in the domain.
  5. Optimization: This final phase aims to improve the access speed of the adapted synonym database. It involves:
  6. Relevance Test: Terms not found in the domain-specific corpus are removed, as they are unlikely to be used.
  7. Redundancy Check: Synsets containing only a single term or duplicate synsets are removed to further reduce the database size without affecting accuracy.

Datasets

  • WordNet: Used as the initial general-purpose synonym database.
  • Aviation Domain Corpus: Specifically, the Aviation Safety Information System (ASRS) corpus (http://asrs.arc.nasa.gov/) was used as the domain-specific corpus for the aviation domain.
  • Query Inventory: A vocabulary of relevant query terms, provided by a user of the IR application, was used for ranking synonymy relations.

Key Insights and Ideas

  • Domain-Specific Synonyms are a Subset: The fundamental idea is that synonymy relations valid in a general context are only a restricted portion of what's relevant in a technical domain.
  • Manual Pruning is Cost-Effective for High-Impact Relations: Focusing manual effort on the most frequently used or impactful synonymy relations yields a good return on investment in terms of precision improvement.
  • Human Agreement is Challenging: The study highlights that even human evaluators can disagree on synonymy judgments, indicating the inherent difficulty and "gray area" in defining domain-specific senses. This underscores the need for automated methods.
  • Automatic Pruning Captures Human Judgment: The correlation between human confidence levels and the scores assigned by the automatic pruning algorithm suggests that the automated method captures significant aspects of human judgment.
  • Optimization Reduces Size Without Losing Accuracy: The optimization phase effectively shrinks the database by removing irrelevant and redundant information, leading to faster access without sacrificing the system's matching accuracy.

Key Results

  • The manual pruning process, involving two human evaluators, showed an agreement rate of 75% on 418 synsets. Disagreements often stemmed from differing strictness in evaluation.
  • The automatic pruning procedure, when tested against a manually checked set, demonstrated a correlation between precision and recall. For example, considering the top 20% of the ranking list yielded 80% precision and 27% recall.
  • The optimization techniques alone, applied to the ASRS corpus, significantly reduced the number of synsets and word-senses. The "Reduced WN" database contained 9,441 synsets and 23,368 word-senses, compared to the "Full WN" with 99,642 synsets and 174,008 word-senses, without loss of accuracy.

Limitations and Future Work

  • Human Agreement Variability: The authors acknowledge the discrepancy in human evaluations, suggesting the task is elusive even for humans. This highlights a limitation in relying solely on manual judgment and the importance of robust automated methods.
  • Completeness Assumption: The study assumes WordNet is complete, meaning it contains all relevant domain-specific senses. It does not check for cases where an appropriate domain-specific sense might be absent from WordNet.
  • Future Work: The paper suggests that while the current approach is promising, there is room for improving the automatic pruning performance. The authors also note that the optimal pruning threshold is a matter for further investigation, depending on whether precision or recall is prioritized.

Summary updated

BibTeX
@inproceedings{turcato2000adapting,
  author = {Davide Turcato and Fred Popowich and Janine Toole and Dan Fass and Devlan Nicholson and Gordon Tisher},
  title = {{Adapting a synonym database to specific domains}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2000},
}
Figure