Improving Word Embeddings Using Kernel PCA

Vishwani Gupta, Sven Giesselbach, Stefan Rüping, Christian BauckhageACL 2019

TL;DR

This paper introduces a novel method to improve word embeddings by "warm-starting" existing models (like Word2Vec and fastText) with morphologically informed embeddings derived from Kernel PCA. This approach significantly boosts performance, especially on smaller datasets and for infrequent words, by better capturing semantic and syntactic relationships.

What Problem Does This Paper Address?

Word embeddings, like those generated by Word2Vec and fastText, are crucial for capturing word meanings and relationships in natural language processing. However, these models typically require massive text corpora to achieve high-quality representations. They struggle with small datasets and often fail to produce good embeddings for infrequent or newly coined words, which is a significant issue in dynamic domains like news. While fastText improves on Word2Vec by incorporating sub-word information, it still faces limitations. This paper aims to address the need for robust word embeddings that perform well even with limited data and can better represent less frequent words by leveraging morphological information.

Methods

The core innovation lies in a two-step process:

  1. Kernel PCA for Morphological Embeddings: The authors first compute a word similarity matrix based on string similarity (e.g., n-gram similarity) for a vocabulary of frequent words. This matrix is then subjected to Kernel PCA (KPCA), a non-linear dimensionality reduction technique. KPCA, applied in a high-dimensional feature space, allows for capturing non-linear relationships, effectively generating "morphologically informed" word embeddings. This step is computationally intensive, so it's applied to a subset of frequent words.
  2. Warm-Starting Existing Models: The KPCA-derived embeddings are then used to "warm-start" or initialize standard Word2Vec (skip-gram) and fastText models. Instead of initializing with random vectors, the models begin with these pre-trained morphological embeddings. This initialization provides a better starting point, allowing the models to learn semantic and syntactic relationships more efficiently, especially when trained on smaller datasets. For fastText, they also compute KPCA vectors for sub-word units.

Datasets

The authors evaluated their models on several datasets:

  • Text8: A large English corpus (10 million words).
  • 20 Newsgroups: A smaller English dataset (1 million words).
  • English Wiki 2016: A very large English corpus (1.192 billion words).
  • German news 2013: A large German corpus (183 million words).

They also used a German version of a semantic/syntactic analogy dataset. The datasets vary in size and domain, allowing for a comprehensive evaluation of the proposed method's robustness.

Key Insights and Ideas

  • Morphology as a "Warm Start": The central idea is that morphological similarity (words sharing roots, affixes) can serve as a strong prior for semantic and syntactic similarity. By explicitly feeding this information into existing models via KPCA, they provide a significant head start.
  • KPCA for Non-Linearity: Using Kernel PCA allows for capturing complex, non-linear relationships in word similarities, which linear PCA might miss. This is particularly beneficial for uncovering subtle morphological patterns.
  • Efficiency on Small Data: The KPCA warm-start approach dramatically improves performance on small datasets, making high-quality embeddings accessible even when large corpora are unavailable.
  • Better Infrequent Word Representation: The method shows promise in generating better embeddings for infrequent words, a known weakness of traditional methods.

Key Results

  • Improved Accuracy: KPCA-initialized skip-gram and fastText models consistently outperformed their baseline counterparts on word similarity and analogy tasks, especially on smaller datasets like Text8 and 20 Newsgroups. For instance, on the Text8 dataset, KPCA fastText achieved higher accuracies than plain fastText in analogy tasks.
  • Faster Convergence: The KPCA-initialized models showed quicker convergence, achieving better results with fewer training epochs compared to cold-start models.
  • Word Similarity: Visualizations (t-SNE) and nearest neighbor lists showed that KPCA models produced more meaningful neighbors, particularly for words like "cruel" and "linguistically," where baselines struggled.
  • Downstream Task Performance: In a sentence classification task using a CNN, models initialized with KPCA embeddings showed improved performance over those initialized with random embeddings, though the margin of improvement was sometimes small.

Limitations and Future Work

  • Computational Cost of KPCA: Computing KPCA on large vocabularies is computationally demanding, which is why the authors restricted it to frequent words.
  • Semantic vs. Syntactic Trade-offs: While KPCA generally improved performance, the paper notes that semantic accuracy sometimes degraded slightly for KPCA fastText and fastText on German analogy tasks, though overall accuracy and syntactic performance improved.
  • Future Directions: The authors suggest exploring different vocabulary selection strategies for KPCA, investigating various string similarity functions, and extending the approach to other embedding models like GloVe, as well as to downstream tasks like Named Entity Recognition and Relation Extraction.

Summary updated

BibTeX
@inproceedings{gupta2019improving,
  author = {Vishwani Gupta and Sven Giesselbach and Stefan Rüping and Christian Bauckhage},
  title = {{Improving Word Embeddings Using Kernel PCA}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure