The More Detail, the Better? – Investigating the Effects of Semantic Ontology Specificity on Vector Semantic Classification with a Plains Cree / nêhiyawêwin Dictionary

Daniel Dacanay, Atticus Harrigan, Arok Wolvengrey, Antti ArppeNAACL 2021

TL;DR

This paper explores whether more detailed semantic ontologies lead to better automatic classification of words in a Plains Cree dictionary. They found that simpler, less specific ontologies like Rapid Words actually produced more human-like classifications than the highly detailed WordNet, suggesting that excessive specificity can hinder vector semantic models.

What Problem Does This Paper Address?

Creating comprehensive semantic dictionaries is a labor-intensive and time-consuming process, often requiring months of manual effort by linguists. While semantic ontologies (hierarchical systems of categories) can guide this process, the question remains: what level of detail in an ontology is optimal for automatically classifying words? This paper investigates whether a more detailed ontology, like Princeton WordNet, leads to better automated semantic classification compared to a less detailed one, like SIL's Rapid Words, when applied to a dictionary of Plains Cree.

Methods

The authors employ vector semantic classification, a technique that represents words as vectors in a multidimensional space based on their co-occurrence in large text corpora. The similarity between words is then calculated using cosine distance. For this study, they used the word2vec tool, leveraging a pre-trained Google News Corpus.

To represent semantic ontologies, they used two main approaches:

  • Princeton WordNet: A well-established, highly detailed, hierarchical semantic ontology.
  • SIL's Rapid Words (RW): A less detailed, hierarchical semantic classification scheme.

They generated vectors for dictionary entries by averaging the vectors of their English glosses. For the ontologies, they generated vectors for synsets (WordNet) and semantic domains (Rapid Words) using their headwords and definitions. These vectors were then compared to the dictionary entry vectors using cosine similarity to rank the semantic classifications.

Datasets

  • Plains Cree / nêhiyawêwin Dictionary: A digitized version of "nêhiyawêwin: itwêwina/Cree: Words (CW)," containing approximately 21,347 words with morphological notes and part-of-speech tags.
  • Google News Corpus: Used for pre-training word2vec embeddings.
  • Princeton WordNet: A widely used semantic ontology.
  • SIL's Rapid Words (RW): A semantic classification scheme.

The paper also refers to manual classifications of the Plains Cree dictionary entries as a "gold standard" for comparison.

Key Insights and Ideas

  • Specificity Paradox: Contrary to intuition, more detailed semantic ontologies (like WordNet) do not necessarily lead to better automated classification. Excessive specificity can overwhelm vector models.
  • The "Regift" Problem: Highly specific, low-frequency words or proper nouns in WordNet synsets can disproportionately influence vector classifications, leading to irrelevant "regift" terms appearing high in rankings. Rapid Words, with its broader categories, significantly mitigates this issue.
  • Domain vs. Synset Granularity: Rapid Words' broader semantic domains, even at a single hierarchical level, proved more effective than WordNet's fine-grained synsets. This suggests that the "umbrella-term" problem is better handled by less granular ontologies.
  • Practical Utility: While not a full replacement for human annotators, vector semantic classification can serve as a valuable preparatory step, significantly reducing the time and effort required for manual semantic annotation.

Key Results

  • Human-likeness: Classifications using Rapid Words were consistently more "human-like" (i.e., manual classifications ranked higher on average) than those using WordNet. For Cree verbs, RW was up to 22 times more accurate in this regard.
  • Top-Ranked Accuracy: With Rapid Words, the top manual classification was found within the top 15 vector-selected classifications 50% of the time for verbs and over 70% of the time for nouns.
  • "Regift" Reduction: The "regift" problem, where irrelevant terms dominate high-ranking classifications, was significantly less pronounced with Rapid Words compared to WordNet.
  • Computational Cost: Cosine comparisons for ~16k CW entries against ~117k WordNet synsets took 4-5 days on a 2-core laptop, while comparisons against Rapid Words domains took about 1.5 days. This can be drastically reduced on computing clusters.

Limitations and Future Work

  • Limitations:
    • The vector method, even with Rapid Words, does not achieve human-like classification as the top choice more than 50% of the time for nouns and less than 30% for verbs.
  • The effectiveness of sentence-level models like BERT is limited when dictionary glosses are non-sentential or overly brief.
  • The study used a specific level (fourth tier) of Rapid Words; further investigation into other levels or using elicitation questions as labels could be beneficial.
  • Future Work:
    • Investigate sentence-based vector generation tools (e.g., BERT) for potentially better polysemy modeling, acknowledging their limitations with brief glosses.
  • Experiment with modifying existing ontologies, such as using higher-level hypernymic synsets in WordNet to reduce specificity.
  • Integrate automatic vector classifications into online dictionaries for user-friendly navigation.

Summary updated

BibTeX
@inproceedings{dacanay2021more,
  author = {Daniel Dacanay and Atticus Harrigan and Arok Wolvengrey and Antti Arppe},
  title = {{The More Detail, the Better? – Investigating the Effects of Semantic Ontology Specificity on Vector Semantic Classification with a Plains Cree / nêhiyawêwin Dictionary}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
}
Figure