Dual-enhanced Word Representations Based on Knowledge Base

Fangyuan He, Yi Zhou, Haodi Zhang, Zhiyong FengISWC (P&D/Industry/BlueSky) 2018 · OpenReview

TL;DR

This paper introduces a novel "dual-enhancement" approach to improve word representations by leveraging large-scale knowledge bases. By treating knowledge base relationships as accurate contextual information and then using mutual reinforcement between strongly associated words, the method significantly boosts performance on synonymy and relatedness tasks, outperforming existing state-of-the-art models.

What Problem Does This Paper Address?

Traditional word representations, like those derived from distributional semantics, rely heavily on large text corpora. While effective, these corpora can be noisy and contain mixed information due to their broad domain coverage. This can hinder the accuracy of word representations, especially for specific tasks. Knowledge bases, on the other hand, offer structured and explicit lexical relationships that can provide more precise contextual information. This paper aims to bridge this gap by integrating knowledge base information to create more accurate and robust word representations.

Methods

The paper proposes a two-layer enhancement strategy:

  1. Knowledge Base as Accurate Context: The first layer injects related words from a knowledge base as additional, accurate context into existing distributional semantics models. This is applied to both neural network-based (skip-gram) and matrix-based (GloVe) approaches. For skip-gram, this involves a modified training objective to maximize the probability of words appearing near a keyword, considering both corpus context and knowledge base neighbors. For GloVe, the co-occurrence matrix is augmented with frequencies derived from knowledge base relationships.
  2. Mutual Reinforcement: The second layer utilizes the knowledge base further by identifying "strongly associated" word pairs (where words are mutually related). These pairs are then used to tune the word representations through mutual reinforcement. This process assigns higher weights to strongly associated words, effectively pulling their representations closer to each other than to more commonly associated words.

Datasets

  • Training Data: A large text corpus (11GB of English Wikipedia) was used for initial word representation training.
  • Knowledge Base: A custom knowledge base was constructed by combining over 155,000 keywords from WordNet and 766,000 from ConceptNet, resulting in approximately 777,000 keywords with related words.
  • Evaluation Datasets:
    • SimLex-999: A well-known benchmark dataset for evaluating word similarity and relatedness.
  • IQ-Synonym-323: A newly constructed dataset for this paper, comprising 323 synonym questions collected from IQ test books and websites. This dataset aims to test human intelligence by assessing synonymy.

Key Insights and Ideas

  • Knowledge Base as a "Noise Filter": Knowledge bases can act as a valuable filter, providing more precise contextual signals to counteract the inherent noise and ambiguity present in large, multi-domain text corpora.
  • Double Enhancement Synergy: The two-stage enhancement process (contextual injection followed by mutual reinforcement) is more effective than either stage alone, demonstrating a synergistic improvement in representation quality.
  • Importance of "Strongly Associated" Words: Explicitly identifying and leveraging mutually related word pairs in the knowledge base for reinforcement is crucial for fine-tuning representations and capturing nuanced semantic relationships.
  • Domain-Specific Datasets Matter: The performance on the newly created IQ-Synonym-323 dataset highlights the need for specialized benchmarks to evaluate word representations in specific semantic tasks, as general benchmarks might not fully capture these nuances.

Key Results

The proposed dual-enhanced approaches (SG-KB-II and GloVe-KB-II) significantly outperformed the baseline models (skip-gram and GloVe) and other state-of-the-art methods on both evaluation datasets.

  • On the IQ-Synonym-323 dataset, SG-KB-II achieved the highest performance at 84.08%, with GloVe-KB-II close behind at 80.85%. This is a substantial improvement over the baseline skip-gram (60.14%) and GloVe (59.61%).
  • On the SimLex-999 dataset, SG-KB-II also performed very well, achieving 0.64, and the first-layer enhancement (SG-KB-I) achieved 0.60.
  • Counter-fitting, a strong baseline, performed exceptionally well on SimLex-999 (0.74) but did not perform as well on the IQ-Synonym-323 dataset (64.75%), underscoring the value of the proposed method for synonymy tasks.

Limitations and Future Work

  • Limitations: The paper does not explicitly discuss limitations in detail. However, the reliance on the quality and coverage of the knowledge base is an inherent limitation. The performance of Counter-fitting on SimLex-999 suggests that while the proposed method excels at synonymy, other methods might have advantages in different types of semantic relationships.
  • Future Work: The authors suggest that their approach can be applied to other distributional semantics models and potentially extended to other NLP tasks where accurate word representations are critical. The availability of the IQ-Synonym-323 dataset also opens avenues for further research and evaluation.

Summary updated

BibTeX
@inproceedings{he2018dual,
  author = {Fangyuan He and Yi Zhou and Haodi Zhang and Zhiyong Feng},
  title = {{Dual-enhanced Word Representations Based on Knowledge Base}},
  booktitle = {ISWC (P\&D/Industry/BlueSky)},
  year = {2018},
  url = {https://openreview.net/forum?id=Pb16Yfeoalg},
}
Figure