Incorporating Trustiness and Collective Synonym/Contrastive Evidence into Taxonomy Construction

Anh Tuan Luu, Jung-jae Kim, See-Kiong NgEMNLP 2015 · OpenReview

Abstract

Taxonomy plays an important role in many applications by organizing domain knowledge into a hierarchy of is-a relations between terms. Previous works on taxonomic relation identification from text corpora lack in two aspects: 1) They do not consider the trustiness of individual source texts, which is important to filter out incorrect relations from unreliable sources. 2) They also do not consider collective evidence from synonyms and contrastive terms, where synonyms may provide additional support to taxonomic relations, while contrastive terms may contradict them. In this paper, we present a method of taxonomic relation identification that incorporates the trustiness of source texts measured with such techniques as PageRank and knowledge-based trust, and the collective evidence of synonyms and contrastive terms identified by linguistic pattern matching and machine learning. The experimental results show that the proposed features can consistently improve performance up to 4%-10% of F-measure.

Introduction

Taxonomies—hierarchical structures of "is-a" relations—are the backbone of structured knowledge. While manual resources like WordNet exist, they are often incomplete for specialized domains like finance or virology. Automating taxonomy construction usually involves scanning text for patterns (e.g., "B such as A"). However, current state-of-the-art methods suffer from three major gaps:

  1. Blind Trust: Most algorithms treat a forum post by a random user with the same weight as a peer-reviewed article. This leads to errors; for instance, the web mistakenly links "American chameleon" more closely to "chameleon" than its actual biological parent, "lizard."
  2. Synonym Fragmentation: Concepts are often expressed in multiple ways. If an algorithm doesn't realize "fruit bat" and "flying fox" are the same thing, it misses out on the "collective evidence" that could confirm their place in a hierarchy.
  3. Ignoring Negative Evidence: Most systems only look for positive matches. They ignore explicit "contrastive" statements in text that clarify two things are not the same, which could serve as a powerful filter for false positives.

Methodology

Building upon a baseline that uses string inclusion, lexical patterns, and syntactic similarity, this research introduces three enhancing modules:

  • Trustiness Scoring: Every data source is assigned a trust score based on four factors: Importance (Google PageRank), Popularity (Alexa Traffic Rank), Authority (Domain type preference for .gov, .mil, and .edu), and Accuracy (a "knowledge-based trust" score verified against Freebase).
  • Collective Synonym Evidence: Synonyms are identified using dictionaries (WordNet, MeSH), pattern matching ("A also known as B"), and a supervised logistic regression model. Evidence scores for all synonymous terms are aggregated to get a robust "collective" score.
  • Contrastive Evidence: The system identifies "negative" patterns like "A is not a type of B" or "A is different from B." This score is used to proportionally decrease the confidence in a taxonomic relation.

Experimental Results

The proposed method was evaluated across six diverse domains: specialized corpora (AI, Finance, and Virus) and web-crawled data (Animals, Plants, and Vehicles). Ground truth was established using WordNet and MeSH.

Performance comparison across Animal and Plant domains showing the impact of individual modules and the combined method.

The inclusion of trust and collective evidence improved the F-measure by 4% to 10% across all domains. For example, in the "Animal" domain, the baseline incorrectly labeled "flying fox" as a type of "fox" due to untrusted forum mentions. The new model correctly filtered this out using contrastive evidence ("flying fox is NOT a fox") and correctly linked it to "bat" by aggregating evidence from its synonym, "fruit bat."

Analysis of Features

One of the most significant insights is the power of "negative" evidence. By explicitly looking for what is not true, the system creates a "sanity check" that significantly boosts precision. Furthermore, the researchers found that trust is multidimensional.

Breakdown of performance improvements for different trustiness and synonym features across four domains.

As shown in the feature breakdown, "Accuracy" (fact-checking) and "Importance" (PageRank) were the most influential trust metrics. Interestingly, a site's local accuracy was often just as strong a signal as its global PageRank. Different features serve different purposes: trustiness improves both precision and recall, synonyms primarily boost recall, and contrastive evidence primarily boosts precision.

Conclusion

This work demonstrates that incorporating source reliability and linguistic relationships like synonymy and contrast significantly improves the accuracy of automated taxonomy construction. Future work may involve automatically discovering more complex linguistic patterns, addressing the "echo chamber" problem where untrusted sites copy each other, and refining trust to be domain-aware.

Summary updated

Abstract

Taxonomy plays an important role in many applications by organizing domain knowledge into a hierarchy of is-a relations between terms. Previous works on the taxonomic relation identification from text corpora lack in two aspects: 1) They do not consider the trustiness of individual source texts, which is important to filter out incorrect relations from unreliable sources. 2) They also do not consider collective evidence from synonyms and contrastive terms, where synonyms may provide additional supports to taxonomic relations, while contrastive terms may contradict them. In this paper, we present a method of taxonomic relation identification that incorporates the trustiness of source texts measured with such techniques as PageRank and knowledge-based trust, and the collective evidence of synonyms and contrastive terms identified by linguistic pattern matching and machine learning. The experimental results show that the proposed features can consistently improve performance up to 4%-10% of F-measure.
BibTeX
@inproceedings{luu2015incorporating,
  author = {Anh Tuan Luu and Jung-jae Kim and See-Kiong Ng},
  title = {{Incorporating Trustiness and Collective Synonym/Contrastive Evidence into Taxonomy Construction}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2015},
  url = {https://openreview.net/forum?id=HJbrtfG_bS},
}
Figure