Complementary Learning of Word Embeddings

Yan Song, Shuming ShiIJCAI 2018 · OpenReview

TL;DR

This paper introduces a novel "complementary learning" framework that jointly trains the Continuous Bag-of-Words (CB) and Skip-Gram (SG) models for word embeddings. By treating CB and SG as agents in a reinforcement learning setup, where the output of one informs the input of the other, they achieve significantly improved word embedding quality compared to training them independently.

What Problem Does This Paper Address?

Traditional methods for learning word embeddings, like the popular Continuous Bag-of-Words (CB) and Skip-Gram (SG) models, operate independently. While both aim to capture semantic relationships by leveraging surrounding words, they tackle the problem from different angles: CB predicts a word given its context, and SG predicts the context given a word. This paper argues that these two approaches are inherently complementary and that their independent training misses an opportunity for mutual enhancement. The challenge is to effectively integrate their strengths to produce higher-quality embeddings, especially in low-resource scenarios where large, high-quality corpora are scarce.

Methods

The core innovation is the "complementary learning" framework, framed as a reinforcement learning problem. Here, CB and SG are treated as two agents learning over a corpus. In each learning iteration:

  1. SG Prediction: An SG model predicts a word (w_sg) given a context. This predicted word is then treated as an intermediate output.
  2. Enlarged Context for CB: The predicted w_sg is appended to the original context, creating an "enlarged context."
  3. CB Prediction: The CB model then uses this enlarged context to predict the target word.
  4. Reward and Update: A reward function is defined based on the performance of both SG (how well it predicts w_sg relative to the original context) and CB (how well it predicts the target word with the enlarged context). The parameters of both SG and CB models are updated using policy gradients based on these rewards.

This iterative process allows information to flow back and forth, with the SG model learning to produce better intermediate words that help the CB model, and the CB model's performance guiding the SG model's updates. The authors also propose a sampling strategy to ensure that the predicted w_sg is not redundant with the existing context.

Datasets

The paper uses the following datasets for training and evaluation:

  • Training Corpora:
    • A large Wikipedia corpus (approximately 2 billion word tokens).
  • A smaller corpus derived from the Wikipedia corpus (1/1000th the size).
  • Intrinsic Evaluation (Word Similarity):
    • MEN-3k
    • Simlex-999
    • WS-353
  • Extrinsic Evaluation (Text Classification):
    • 20Newsgroups (20NG) for topic classification.
  • ATIS for intent classification.
  • TREC for question type classification.
  • IMDB for sentiment classification.
  • Retrofitting Evaluation:
    • PPDB (Paraphrase Database) is used as an external semantic source for retrofitting.

Key Insights and Ideas

  • Complementarity is Key: CB and SG are not just alternative methods but complementary tasks that can mutually benefit from each other's outputs.
  • Reinforcement Learning as a Unifying Framework: Framing the interaction as a reinforcement learning problem provides a flexible and principled way to combine the two models and manage the flow of information.
  • SG's Role in Enhancing CB: The intermediate word predicted by SG can provide valuable additional context for the CB model, leading to better predictions.
  • Pre-trained Embeddings Matter: Starting with pre-trained embeddings significantly boosts the performance of the complementary learning approach, especially in the initial iterations.
  • Effective Domain Adaptation and Retrofitting: The framework naturally lends itself to domain adaptation and retrofitting, improving embeddings by incorporating external knowledge or in-domain context.

Key Results

  • Improved Word Similarity: The proposed complementary learning models (CTCB and CTSG) consistently outperformed individual CB and SG models, as well as other baselines like GloVe and full-gradient approaches, across various word similarity benchmarks (MEN-3k, Simlex-999, WS-353). For instance, on the large corpus, CTSGP achieved a Spearman's rank correlation of 67.00 on WS-353, outperforming others.
  • Superior Classification Performance: In extrinsic evaluations on text classification tasks (20NG, ATIS, TREC, IMDB), the complementary learned embeddings also showed significant improvements. For example, on the 20NG dataset, CTCBT-T achieved 83.49%, outperforming the baseline CBT (82.82%) and other methods.
  • Effective Retrofitting: When used as a retrofitting method with PPDB, the complementary learning approach significantly improved word similarity scores compared to previous state-of-the-art retrofitting techniques.
  • Robustness to Corpus Size: The effectiveness of the complementary learning approach was observed on both large and small corpora, with improvements on the smaller corpus being somewhat larger, suggesting its utility in low-resource settings.

Limitations and Future Work

  • Computational Overhead: The complementary learning process is slightly longer than training individual CB or SG models due to the intermediate word generation step.
  • Future Directions: The authors suggest extending this work to learning embeddings for knowledge graphs and exploring its application in multilingual and cross-lingual settings. They also highlight the potential for using this framework to incorporate other forms of external knowledge.

Summary updated

Abstract

Continuous bag-of-words (CB) and skip-gram (SG) models are popular approaches to training word embeddings. Conventionally they are two standing-alone techniques used individually. However, with the same goal of building embeddings by leveraging surrounding words, they are in fact a pair of complementary tasks where the output of one model can be used as input of the other, and vice versa. In this paper, we propose complementary learning of word embeddings based on the CB and SG model. Specifically, one round of learning first integrates the predicted output of a SG model with existing context, then forms an enlarged context as input to the CB model. Final models are obtained through several rounds of parameter updating. Experimental results indicate that our approach can effectively improve the quality of initial embeddings, in terms of intrinsic and extrinsic evaluations.
BibTeX
@inproceedings{song2018complementary,
  author = {Yan Song and Shuming Shi},
  title = {{Complementary Learning of Word Embeddings}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2018},
  url = {https://openreview.net/forum?id=BkZhx4MO-r},
}
Figure