Incorporating Extra Knowledge to Enhance Word Embedding

Arpita Roy, Shimei PanIJCAI 2020 · DOI

TL;DR

This survey paper explores how to enrich word embeddings, the numerical representations of words, by incorporating external knowledge sources. It highlights that traditional embeddings, while useful, often miss nuanced semantic relationships and struggle with rare words, and this survey provides a comprehensive overview of recent methods to address these limitations.

What Problem Does This Paper Address?

Word embeddings are fundamental to many Natural Language Processing (NLP) tasks, enabling machines to understand and process text. These embeddings represent words as dense vectors in a continuous space, capturing semantic relatedness based on word co-occurrence in large text corpora. However, this approach has limitations:

  • Limited Semantic Nuance: While embeddings can tell us that "kitten" and "cat" are related, they struggle to explicitly encode the type of relationship (e.g., "kitten is a juvenile cat").
  • Polysemy: A single word can have multiple meanings (e.g., "bank" as a financial institution or a river's edge), and traditional embeddings assign a single vector, failing to distinguish these senses.
  • Data Sparsity: Rare words, which appear infrequently in training data, often have unreliable embeddings.

To overcome these challenges, researchers are exploring ways to inject "extra knowledge" into word embedding models. This survey aims to organize and summarize these efforts.

Methods

The paper categorizes the methodologies for incorporating knowledge into word embeddings along four key dimensions:

  • Static vs. Dynamic Embedding:
    • Static: Learns a single, context-independent representation for each word (e.g., Word2Vec). Most surveyed papers focus on enhancing these.
  • Dynamic/Contextual: Generates representations that vary with the word's context (e.g., BERT, ELMO). This is a newer area with growing research.
  • Joint Optimization vs. Post-Processing:
    • Joint Optimization: Word embeddings are trained simultaneously with the knowledge source, modifying the training objective to incorporate both.
  • Post-Processing: External knowledge is used to refine pre-trained word embeddings, often through methods like retrofitting. This is beneficial as it doesn't require expensive retraining.
  • Solo vs. Coupled Embedding:
    • Solo: Only word embeddings are learned.
  • Coupled: Word embeddings are trained alongside other embeddings (e.g., sense embeddings, entity embeddings) in the same space.
  • Single vs. Multi-Sense Embedding:
    • Single-Sense: Assigns one vector per word, ignoring polysemy.
  • Multi-Sense: Aims to represent different meanings of a word with distinct vectors, often requiring external sense inventories for static models or leveraging context for dynamic models.

The paper also discusses various knowledge types that can be incorporated: textual (from Wikipedia, dictionaries), morphological (roots, affixes), categorical (semantic types, POS tags), relational (triplets like head-relation-tail), and knowledge graphs.

Datasets

The paper does not explicitly list specific datasets used for training or evaluation in a dedicated section. However, it mentions various knowledge sources that can be considered as data for enriching embeddings:

  • Wikipedia: A large source of textual descriptions and interlinked articles.
  • WordNet & BabelNet: Lexical databases encoding semantic relations.
  • Freebase & Probase: Structured data repositories of entities and relations.
  • Domain Dictionaries: Specialized vocabularies like UMLS (biomedical) and MAEC (malware).
  • Other NLP Tools: Outputs from tools like Morfessor (morphological analysis) and LDA (topic modeling).
  • PPDB: Paraphrase pairs.
  • General Dictionaries: Oxford, Cambridge, Collins.
  • Social Graphs: Representing relationships between users.
  • Human Annotations: For word properties or relations.

Key Insights and Ideas

  • Knowledge Injection is Crucial: Traditional word embeddings, while powerful, are limited by their reliance solely on co-occurrence. Incorporating external knowledge is key to capturing deeper semantic nuances, handling polysemy, and improving representations for rare words.
  • Diverse Knowledge Sources: The effectiveness of knowledge injection depends on the type of knowledge used. Morphological, syntactic, semantic, and even domain-specific knowledge can all contribute to better embeddings.
  • Static vs. Dynamic Trade-offs: While static embeddings are more computationally accessible, dynamic embeddings (like BERT) are showing superior performance by capturing context. However, integrating knowledge with dynamic models is more computationally demanding.
  • Post-processing Offers Flexibility: Methods that refine pre-trained embeddings (post-processing) offer a practical advantage by avoiding costly retraining and can be applied to any existing embedding model.
  • Multi-Sense Embeddings are Promising: Addressing word polysemy through multi-sense embeddings significantly improves performance, especially when external sense information is leveraged.

Key Results

The paper is a survey, so it doesn't present its own experimental results. Instead, it summarizes the findings of numerous prior works. The key takeaway from these works is that incorporating external knowledge generally leads to improvements in downstream NLP tasks. For instance, the survey notes that multi-sense embeddings empirically show significantly better performance over single-sense embeddings. The paper also highlights that the majority of surveyed papers (20 out of 22) focus on enhancing static word embeddings, with a recent shift towards dynamic embeddings.

Limitations and Future Work

The authors identify several limitations and suggest future research directions:

  • Understanding Impact: There's a limited understanding of how adding knowledge affects the inherent properties of word embeddings, such as their linear behavior and compositionalality. Future work should focus on theory-guided methods and visualization techniques to better interpret these impacts.
  • Diverse Knowledge Types: Most research focuses on linguistic knowledge. Incorporating other forms, like First Order Logic (FOL), which is powerful for representing human knowledge, is an underexplored area.
  • Corpus-Level Constraints: There's a need to incorporate constraints that ensure desirable properties in embeddings, such as fairness. For example, addressing human biases present in text data by encoding fairness as a corpus-level constraint is a promising direction.
  • Efficiency: Learning word embeddings, especially contextual ones, is computationally expensive. Adding knowledge further exacerbates this. Future research needs to balance accuracy improvements with computational efficiency, making these methods accessible to a wider range of researchers.

Summary updated

BibTeX
@inproceedings{roy2020incorporating,
  author = {Arpita Roy and Shimei Pan},
  title = {{Incorporating Extra Knowledge to Enhance Word Embedding}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.24963/ijcai.2020/686},
}
Figure