Single Training Dimension Selection for Word Embedding with PCA

Yu WangEMNLP 2019 · DOI

TL;DR

This paper introduces a fast and reliable method using Principal Component Analysis (PCA) to automatically select the optimal number of dimensions for word embeddings. By training a single, high-dimensional embedding and then incrementally reducing dimensions while monitoring performance, the authors demonstrate a way to achieve competitive accuracy with significantly fewer dimensions, leading to smaller models and faster training.

What Problem Does This Paper Address?

Word embeddings are crucial for many Natural Language Processing (NLP) tasks, representing words as dense vectors in a continuous space. The dimensionality of these vectors is a critical hyperparameter: too few dimensions lead to underfitting, while too many can cause overfitting and increase model size and computational cost. While common dimensionalities like 300 are often used due to historical precedent, there's a lack of a principled method to determine the optimal dimensionality for a given dataset and task. This paper aims to fill that gap by providing an efficient way to find this sweet spot, balancing model performance with size.

Methods

The core of the proposed method is a PCA-based approach.

  1. Initial High-Dimensional Training: A word embedding model (Skip-gram in their experiments) is trained with a generous upper bound on dimensions (e.g., 1000).
  2. PCA Transformation: The trained high-dimensional embeddings are then transformed using PCA. This step essentially re-orients the embedding space to align with the directions of maximum variance.
  3. Incremental Dimension Reduction: Starting from the highest dimension, dimensions are incrementally removed one by one. After each removal, the performance of the resulting lower-dimensional embeddings is evaluated on a chosen language task (e.g., word analogy or similarity).
  4. Optimal Dimension Selection: The method identifies the dimension where performance is maximized while considering the trade-off between model size and accuracy. The authors frame this as finding a "Pareto-optimal dimension."

This approach is novel in that it avoids training multiple embeddings from scratch, which is the standard for grid search, and instead leverages PCA to efficiently explore the dimensionality space.

Datasets

The paper uses the following datasets:

  • Training Datasets: Text8 and WikiText-103.
  • Evaluation Datasets (for similarity tasks): Wordsim 353, RW Stanford, and MTurk 771.

No new benchmarks were introduced; existing, widely-used datasets were employed for training and evaluation.

Key Insights and Ideas

  • PCA as a Dimensionality Reduction Tool for Embeddings: The paper shows that PCA can effectively identify and discard dimensions that contribute little to the explained variance in the embeddings, without significantly harming performance on downstream tasks.
  • Bias-Variance Trade-off in Dimension Selection: The authors draw an analogy to the bias-variance trade-off, suggesting that reducing dimensions increases bias but decreases variance, and their method helps find the optimal balance.
  • Dimensionality is Task-Dependent but Consistent: While the optimal dimensionality might vary slightly across tasks, the method consistently identifies a similar optimal dimension regardless of the initial upper bound chosen (e.g., 500 vs. 1000), as long as the upper bound is sufficiently large.
  • Efficiency Gains: The PCA-based approach offers substantial speedups compared to traditional grid search for dimensionality selection.

Key Results

  • Accuracy: The proposed PCA-based method, after retraining the embedding with the selected optimal dimension, achieves performance competitive with embeddings found through exhaustive grid search. On average, it achieves 100.2% (Wordsim 353) to 96.9% (MTurk 771) of the optimal performance found by grid search.
  • Efficiency: The method provides a significant speedup. For Text8, it offers a 13.1x speedup, and for WikiText-103, a 12.7x speedup compared to grid search. The PCA step itself is very fast (22-34 minutes).
  • Robustness: The selected optimal dimensionality is consistent across different upper bounds (e.g., 500 to 1000), indicating the method's robustness.
  • Dimensionality Reduction: The method can identify optimal dimensions that are significantly lower than the initial upper bound, leading to smaller embedding sizes. For example, for Text8, the optimal dimension identified was 34 for Wordsim 353, 1 for RW Stanford, and 65 for MTurk 771.

Limitations and Future Work

  • Retraining Required: While the selection of the optimal dimension is fast, achieving the final competitive performance requires retraining the embedding with the selected dimension.
  • Upper Bound Selection: While the method is robust to the choice of upper bound, a sufficiently large upper bound is still necessary. The paper doesn't provide explicit guidance on how to choose this bound for novel datasets.
  • Future Work: The authors suggest that this method can be extended to other embedding algorithms and potentially to other dimensionality reduction techniques beyond PCA. They also highlight the potential for this method in model compression.

Summary updated

BibTeX
@inproceedings{wang2019single,
  author = {Yu Wang},
  title = {{Single Training Dimension Selection for Word Embedding with PCA}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  doi = {10.18653/v1/d19-1369},
}
Figure