Learning Term Embeddings for Lexical Taxonomies

Jingping Liu, Menghui Wang, Chao Wang, Jiaqing Liang, Lihan Chen, Haiyun Jiang, Yanghua Xiao, Yunwen Chen

Abstract

Lexical Taxonomies (LTs) are structured knowledge bases that capture hierarchical "is-a" relations between terms. While Knowledge Graph embedding is a mature field, standard techniques often struggle with the unique structural properties of LTs, such as strict hierarchy and term polysemy. This paper introduces a specialized embedding method that incorporates asymmetric term representations, an attention mechanism to handle polysemy, and implicit sibling relations. Experimental results on the Probase dataset demonstrate that the proposed model significantly outperforms existing baselines and provides a performance boost to pre-trained language models like BERT.

Introduction

Lexical Taxonomies (LTs) represent semantic hierarchies through hyponym-hypernym relations (e.g., "apple" is a hyponym of "fruit"). Unlike general Knowledge Graphs, LTs are characterized by a strict directional hierarchy and significant term polysemy. Traditional embedding models like TransE or RotatE, designed for multi-relational graphs, often fail to capture these nuances, leading to hierarchical collapse or semantic noise.

Challenges in Lexical Taxonomy Embedding

The authors identify three primary hurdles in current embedding methods:

  1. Hierarchical Contradiction: Translation-based models often collapse the hierarchy mathematically when multiple "is-a" relations are chained.
  2. Implicit Sibling Relations: Semantic similarity between siblings (e.g., "dog" and "cat") is vital but often missing from explicit relations in sparse taxonomies.
  3. Polysemy: Terms with multiple meanings introduce noise. For instance, the term "tank" can refer to a storage container or a military vehicle, leading to conflicting training signals if treated as a single point.
Figure 2: Polysemous terms like "tank" can belong to vastly different semantic branches, creating noise in standard embedding spaces.

Methodology

To address these issues, the proposed model learns two distinct vectors for each term: a hyponym embedding ($v_t$) and a hypernym embedding ($u_t$). This asymmetry naturally models the directional nature of taxonomies. Key innovations include:

  • Term-level Attention Mechanism: An attention layer filters polysemous noise by downweighting hyponyms that do not align with a specific hypernym's semantic context.
  • Random Walk Sibling Detection: A random walk process identifies implicit siblings, providing a denser training signal for terms that share common hypernyms and hyponyms.
  • Curriculum Learning: Training follows a "typicality" ranking based on text co-occurrence, starting with common, unambiguous relations before tackling rarer, more complex cases.

Experimental Results

The model was evaluated on two subsets of the Probase dataset: Pro5K and Pro300K. As shown in the results below, the proposed "HASC" configuration (incorporating Hyponym/Hypernym vectors, Attention, Siblings, and Curriculum learning) achieves the highest accuracy across both subsets, outperforming specialized hierarchical models like On2Vec.

Figure 6: Performance comparison on Probase datasets. The proposed HASC model consistently outperforms standard KG embedding baselines in relation classification.

Beyond link prediction and classification, the learned embeddings were used to augment BERT and RoBERTa for text classification on the AG's News dataset. Concatenating these structured embeddings led to a consistent performance lift (e.g., BERT accuracy rising from 92.0% to 93.4%), suggesting that LTs contain valuable structured knowledge that even massive language models haven't fully captured during pre-training.

Conclusion

By explicitly modeling the unique constraints of lexical taxonomies—specifically polysemy and sibling relations—the proposed method generates superior term representations. This structured knowledge effectively complements pre-trained transformers, offering a path toward more robust NLP applications. Future work aims to integrate richer knowledge graph attributes, such as entity-attribute-value information, into the embedding process.

Summary updated

Abstract

Lexical taxonomies, a special kind of knowledge graph, are essential for natural language understanding. This paper studies the problem of lexical taxonomy embedding. Most existing graph embedding methods are difficult to apply to lexical taxonomies since 1) they ignore implicit but important information, namely, sibling relations, which are not explicitly mentioned in lexical taxonomies and 2) there are lots of polysemous terms in lexical taxonomies. In this paper, we propose a novel method for lexical taxonomy embedding. This method optimizes an objective function that models both hyponym-hypernym relations and sibling relations. A term-level attention mechanism and a random walk based metric are then proposed to assist the modeling of these two kinds of relations, respectively. Finally, a novel training method based on curriculum learning is proposed. We conduct extensive experiments on two tasks to show that our approach outperforms other embedding methods and we use the learned term embeddings to enhance the performance of the state-of-the-art models that are based on BERT and RoBERTa on text classification.
BibTeX
@inproceedings{liu2021learning,
  author = {Jingping Liu and Menghui Wang and Chao Wang and Jiaqing Liang and Lihan Chen and Haiyun Jiang and Yanghua Xiao and Yunwen Chen},
  title = {{Learning Term Embeddings for Lexical Taxonomies}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i7.16795},
}
Figure