Contextual Skipgram: Training Word Representation Using Context Information

Dongjae Kim, Jong-Kook KimarXiv 2021 · arXiv · OpenReview

TL;DR

This paper introduces "Contextual Skip-gram" (CSG), an enhancement to the standard Skip-gram word embedding model. CSG improves word representation quality by explicitly considering the semantic relevance of context words, rather than treating all surrounding words equally, leading to better performance on various NLP tasks.

What Problem Does This Paper Address?

Traditional Skip-gram (SG) models learn word embeddings by predicting surrounding words given a center word. However, the context window often contains words that are not semantically relevant to the center word (e.g., common words, function words). These "weak co-occurrences" can negatively impact the learning process, potentially hindering the model from capturing nuanced semantic relationships and leading to degraded performance, especially as training progresses. The paper aims to address this by developing a method that can better distinguish and leverage the importance of different context words.

Methods

The core innovation is the Contextual Skip-gram (CSG) model. Unlike standard SG, CSG incorporates "context information" into its prediction mechanism. This context information is derived from the surrounding words within the sliding window.

Key components of CSG include:

  • Context Function: This function aggregates the embeddings of surrounding words into a single "context embedding" ($v_{con}$). The paper mentions averaging as a simple approach but opts for a summing function for computational efficiency.
  • Weighted Fusion: CSG combines the prediction probabilities derived from the center word alone and from the context embedding. Two fusion strategies are proposed:
  • Early Fusion (EF): The center word vector and context vector are element-wise weighted and summed before calculating the probability.
  • Late Fusion (LF): The probabilities are calculated separately for the center word and context, and then these probabilities are weighted and summed.
  • Fusion Weight ($\gamma$): A hyperparameter ($\gamma$) controls the balance between the center word's influence and the context information's influence. This weight can be static or dynamic (e.g., linearly increasing over epochs).

The paper also details how the gradient updates are modified to account for the context vector, aiming to preserve training speed despite the added complexity.

Datasets

The paper uses the following datasets:

  • Training Corpora:
    • A large corpus extracted from Wikipedia (approx. 4.14 billion tokens).
  • A smaller corpus created by 1% random sampling of sentences from the Wikipedia corpus (approx. 39.3 million tokens).
  • The text8 corpus (mentioned in Appendix A.1).
  • Evaluation Benchmarks:
    • Word Similarity: Simlex-999, WordSim-353, and MEN-3k datasets.
  • Word Analogy: Google analogy dataset and MSR analogy dataset.
  • Named Entity Recognition (NER): CoNLL-2003 English dataset.

Key Insights and Ideas

  • Not all context is equal: The paper's central premise is that semantic relevance of context words varies, and standard SG treats them uniformly, which is suboptimal.
  • Contextual information improves representation: Explicitly modeling and leveraging the context embedding leads to more robust and semantically richer word representations.
  • Fusion strategy matters: Both Early Fusion (EF) and Late Fusion (LF) show improvements, with LF performing slightly better on smaller corpora and EF on larger ones.
  • Fusion weight tuning is crucial: A higher fusion weight ($\gamma$) generally leads to better performance on similarity tasks, suggesting that context information is highly valuable. However, extreme values (like $\gamma=1$) can sometimes degrade performance, indicating a need for a balanced approach.
  • Dynamic fusion weights can help: The linear fusion scheme ($\gamma_{linear,0\to1}$) achieved superior overall scores, suggesting that adaptively increasing the reliance on context during training can be beneficial.
  • Task-dependent performance: While CSG excels in word similarity and analogy tasks, its benefits are less pronounced or even degraded in tasks like Named Entity Recognition, suggesting that the utility of context information is task-specific.

Key Results

  • Word Similarity: CSG models consistently outperformed baseline models (CBOW, SG, DSG, SSSG, FastText) on small and large corpora across Simlex-999, WordSim-353, and MEN-3k datasets. For instance, on the large corpus, CSG models achieved scores up to 68.29 (Sim-999), 73.16 (WS-353), and 76.95 (MEN-3k), compared to SG's 71.65, 74.81, and 73.53 respectively.
  • Word Analogy: CSG models with high $\gamma$ values and the linear fusion scheme showed strong performance, with CSGEF,0→1 achieving the best overall result on the Google and MSR analogy tasks.
  • Named Entity Recognition: Contrary to other tasks, CSG models with high $\gamma$ showed degraded performance compared to SG. However, low context weights and the random fusion scheme (Yran) performed comparably or better than the original SG model.
  • Prediction Analysis: CSG showed improved prediction of surrounding words, especially for less relevant words that were poorly predicted by SG.

Limitations and Future Work

  • Computational Cost: The paper acknowledges that additional computation and word vector updates proportional to the window size could potentially harm training speed.
  • Task Specificity: The performance of CSG varies across tasks, with NER being a notable example where its benefits are not as clear.
  • Future Work: The authors suggest further research into more sophisticated context functions and fusion weight schemes to potentially unlock even better performance. They also note that the CSG model could be further improved by incorporating external linguistic resources.

Summary updated

Abstract

The skip-gram (SG) model learns word representation by predicting the words surrounding a center word from unstructured text data. However, not all words in the context window contribute to the meaning of the center word. For example, less relevant words could be in the context window, hindering the SG model from learning a better quality representation. In this paper, we propose an enhanced version of the SG that leverages context information to produce word representation. The proposed model, Contextual Skip-gram, is designed to predict contextual words with both the center words and the context information. This simple idea helps to reduce the impact of irrelevant words on the training process, thus enhancing the final performance
BibTeX
@misc{kim2021contextual,
  author = {Dongjae Kim and Jong-Kook Kim},
  title = {{Contextual Skipgram: Training Word Representation Using Context Information}},
  year = {2021},
  eprint = {2102.08565},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2102.08565},
}
Figure