TL;DR
This paper introduces a novel "complementary learning" framework that jointly trains the Continuous Bag-of-Words (CB) and Skip-Gram (SG) models for word embeddings. By treating CB and SG as agents in a reinforcement learning setup, where the output of one informs the input of the other, they achieve significantly improved word embedding quality compared to training them independently.
What Problem Does This Paper Address?
Traditional methods for learning word embeddings, like the popular Continuous Bag-of-Words (CB) and Skip-Gram (SG) models, operate independently. While both aim to capture semantic relationships by leveraging surrounding words, they tackle the problem from different angles: CB predicts a word given its context, and SG predicts the context given a word. This paper argues that these two approaches are inherently complementary and that their independent training misses an opportunity for mutual enhancement. The challenge is to effectively integrate their strengths to produce higher-quality embeddings, especially in low-resource scenarios where large, high-quality corpora are scarce.
Methods
The core innovation is the "complementary learning" framework, framed as a reinforcement learning problem. Here, CB and SG are treated as two agents learning over a corpus. In each learning iteration:
- SG Prediction: An SG model predicts a word (
w_sg) given a context. This predicted word is then treated as an intermediate output. - Enlarged Context for CB: The predicted
w_sgis appended to the original context, creating an "enlarged context." - CB Prediction: The CB model then uses this enlarged context to predict the target word.
- Reward and Update: A reward function is defined based on the performance of both SG (how well it predicts
w_sgrelative to the original context) and CB (how well it predicts the target word with the enlarged context). The parameters of both SG and CB models are updated using policy gradients based on these rewards.
This iterative process allows information to flow back and forth, with the SG model learning to produce better intermediate words that help the CB model, and the CB model's performance guiding the SG model's updates. The authors also propose a sampling strategy to ensure that the predicted w_sg is not redundant with the existing context.
Datasets
The paper uses the following datasets for training and evaluation:
- Training Corpora:
- A large Wikipedia corpus (approximately 2 billion word tokens).
- A smaller corpus derived from the Wikipedia corpus (1/1000th the size).
- Intrinsic Evaluation (Word Similarity):
- MEN-3k
- Simlex-999
- WS-353
- Extrinsic Evaluation (Text Classification):
- 20Newsgroups (20NG) for topic classification.
- ATIS for intent classification.
- TREC for question type classification.
- IMDB for sentiment classification.
- Retrofitting Evaluation:
- PPDB (Paraphrase Database) is used as an external semantic source for retrofitting.
Key Insights and Ideas
- Complementarity is Key: CB and SG are not just alternative methods but complementary tasks that can mutually benefit from each other's outputs.
- Reinforcement Learning as a Unifying Framework: Framing the interaction as a reinforcement learning problem provides a flexible and principled way to combine the two models and manage the flow of information.
- SG's Role in Enhancing CB: The intermediate word predicted by SG can provide valuable additional context for the CB model, leading to better predictions.
- Pre-trained Embeddings Matter: Starting with pre-trained embeddings significantly boosts the performance of the complementary learning approach, especially in the initial iterations.
- Effective Domain Adaptation and Retrofitting: The framework naturally lends itself to domain adaptation and retrofitting, improving embeddings by incorporating external knowledge or in-domain context.
Key Results
- Improved Word Similarity: The proposed complementary learning models (CTCB and CTSG) consistently outperformed individual CB and SG models, as well as other baselines like GloVe and full-gradient approaches, across various word similarity benchmarks (MEN-3k, Simlex-999, WS-353). For instance, on the large corpus, CTSGP achieved a Spearman's rank correlation of 67.00 on WS-353, outperforming others.
- Superior Classification Performance: In extrinsic evaluations on text classification tasks (20NG, ATIS, TREC, IMDB), the complementary learned embeddings also showed significant improvements. For example, on the 20NG dataset, CTCBT-T achieved 83.49%, outperforming the baseline CBT (82.82%) and other methods.
- Effective Retrofitting: When used as a retrofitting method with PPDB, the complementary learning approach significantly improved word similarity scores compared to previous state-of-the-art retrofitting techniques.
- Robustness to Corpus Size: The effectiveness of the complementary learning approach was observed on both large and small corpora, with improvements on the smaller corpus being somewhat larger, suggesting its utility in low-resource settings.
Limitations and Future Work
- Computational Overhead: The complementary learning process is slightly longer than training individual CB or SG models due to the intermediate word generation step.
- Future Directions: The authors suggest extending this work to learning embeddings for knowledge graphs and exploring its application in multilingual and cross-lingual settings. They also highlight the potential for using this framework to incorporate other forms of external knowledge.
Summary updated