Reverse Transfer Learning: Can Word Embeddings Trained for Different NLP Tasks Improve Neural Language Models?

Lyan Verwimp, Jerome R. BellegardaInterspeech 2019

TL;DR

This paper explores "reverse transfer learning" in NLP, investigating if word embeddings trained on tasks other than language modeling can improve neural language models (LMs). They found that pre-trained embeddings can indeed boost LM performance, but only when the pre-training task and data closely align with the LM's objective, suggesting that task congruence is key for effective transfer.

What Problem Does This Paper Address?

Traditional transfer learning in NLP typically involves using a language model (LM) trained on a massive, self-annotated dataset to improve performance on other NLP tasks that might have limited labeled data. This works because LMs learn rich contextual representations. However, this paper asks the inverse question: can we leverage representations trained for different NLP tasks to improve the LM itself? The authors hypothesize that LMs, primarily optimized for predicting the next word, might overlook crucial global semantic information present in a document. They propose that embeddings trained on tasks that capture this global context could help LMs make better predictions by providing a richer understanding of the text's overall meaning.

Methods

The core of the paper is an experimental investigation into different types of pre-trained embeddings for neural LMs. They categorize pre-trained embeddings into two main types:

  1. Embeddings closely related to language modeling: This includes standard word2vec embeddings, which are trained on local word contexts, and embeddings derived from bidirectional LSTMs trained as LMs.
  2. Embeddings promoting global semantic information: This category includes embeddings from a bidirectional LSTM trained as a domain classifier. The idea here is that by forcing the LSTM to predict the overall semantic domain of a paragraph, its internal states (used as embeddings) would capture broader contextual meaning.

They compare these pre-trained embeddings against a baseline LM trained from scratch. Crucially, they keep the pre-trained embedding weights fixed during LM training to isolate the effect of the pre-trained representations. They also explore averaging cell states from bidirectional LSTMs to create context-independent embeddings from context-dependent models.

Datasets

  • LM Training Data: An in-house dataset of 10 million words with a vocabulary of 25,000 words, described as diverse in domain and style.
  • Word Embedding Training Data: A larger superset of 5 billion words, containing the same mixture of domains and styles as the LM training data.
  • Domain Classifier Training Data:
    • A "small" dataset of 7 million words, and a "large" dataset of 70 million words, both containing paragraphs labeled with semantic domains. The domain labels were generated through an iterative process involving LSA and k-means clustering.
  • External Word Embeddings: Freely available CBOW embeddings trained on a 100 billion word dataset from Google News (300-dimensional).

Key Insights and Ideas

  • "Reverse Transfer Learning" is Viable: The paper demonstrates that it's possible to improve LMs by using embeddings trained on other tasks, challenging the unidirectional flow of traditional transfer learning.
  • Task Congruence is Paramount: The most significant finding is that the effectiveness of pre-trained embeddings heavily depends on how closely their training task and data align with the target LM task. Embeddings trained on similar objectives (like word2vec or bidirectional LMs) and similar data (sentences) yielded better results than those trained on vastly different tasks (like domain classification) or data formats.
  • Global Context Matters, But How to Capture It? While the hypothesis that global context is important for LMs is supported, directly using domain classifier embeddings didn't yield the expected improvements. This suggests that simply forcing a model to predict a global label might not translate directly into useful representations for next-word prediction.
  • Bidirectional LSTMs for Embeddings: Using embeddings from bidirectional LSTMs, even when the final LM is unidirectional, showed promise, suggesting that capturing richer contextual information during pre-training can be beneficial.

Key Results

The paper presents results in terms of perplexity (PPL), where lower is better.

  • Baseline LM: Perplexity of 189.
  • word2vec on 5B sentences: Achieved a perplexity of 162, a significant 14% relative reduction.
  • Google News word2vec (100B words): Perplexity of 195, showing no improvement over the baseline, likely due to data mismatch.
  • Domain Classifier Embeddings (7M paragraphs): Perplexity of 255, worse than the baseline.
  • Domain Classifier Embeddings (7M sentences): Perplexity of 239, still worse than baseline but better than paragraph-based domain embeddings.
  • Bidirectional LM Embeddings (10M LM data): Achieved a perplexity of 185, a small but notable improvement over the baseline.

The results strongly suggest that word2vec embeddings trained on a large, sentence-based dataset aligned with the LM's task performed best among the tested methods.

Limitations and Future Work

  • Data Mismatch: The authors repeatedly highlight data mismatch as a key reason for the poor performance of domain classification embeddings. The format (paragraphs vs. sentences) and the nature of the training objective (global domain vs. local prediction) seem to be critical factors.
  • Saturation of Domain Embeddings: For domain embeddings trained on paragraphs, they observed saturated values, suggesting potential issues with how the LSTM states were accumulating information over long sequences. While they experimented with solutions, significant improvements weren't achieved.
  • Future Work: The authors suggest exploring multi-task learning objectives where language modeling is combined with predicting a semantic label. This could potentially help LMs better integrate global semantic information while still optimizing for next-word prediction. They also note that further improvements might be seen with larger datasets for bidirectional LM embeddings.

Summary updated

BibTeX
@inproceedings{verwimp2019reverse,
  author = {Lyan Verwimp and Jerome R. Bellegarda},
  title = {{Reverse Transfer Learning: Can Word Embeddings Trained for Different NLP Tasks Improve Neural Language Models?}},
  booktitle = {Interspeech},
  year = {2019},
}
Figure