Improving Neural Machine Translation with Compact Word Embedding Tables

Krtin Kumar, Mehdi Rezagholizadeh, Yiu Sing Lau, Qun Liu

TL;DR

This paper explores reducing the memory footprint of neural machine translation (NMT) models by using compressed or even random word embeddings. They demonstrate that NMT models can maintain high performance with significantly smaller embedding tables, sometimes even outperforming baselines, by using a novel Gaussian Product Quantization (GPQ) method.

What Problem Does This Paper Address?

Neural machine translation (NMT) models rely heavily on embedding matrices to represent words and sub-word units (tokens) as numerical vectors. These embedding matrices can be quite large, consuming a significant portion of a model's total parameters. For instance, in a Transformer model with a vocabulary of 50,000 tokens, embedding matrices can account for 36% of the model's parameters. This large memory requirement poses a challenge for deploying NMT systems on resource-constrained devices (like mobile phones) or in industrial settings where efficiency is paramount. The paper aims to investigate how much of the information in these large embedding tables is truly necessary for NMT performance and to develop methods for creating more compact embedding representations without sacrificing too much accuracy.

Methods

The authors propose two main approaches:

  • Random Word Embeddings (RWE): This is the most aggressive compression strategy, where word embeddings are initialized with completely random values sampled from a Gaussian distribution. The idea is to see how much performance degrades when no pre-learned semantic or syntactic information is present. They also include an optional linear transformation layer to help the model adapt to these random embeddings.
  • Gaussian Product Quantization (GPQ): This is a novel method that builds upon Product Quantization (PQ). PQ compresses embedding matrices by partitioning them into groups and then clustering the vectors within each group. GPQ extends this by modeling each cluster with a Gaussian distribution, incorporating both the cluster's mean and its variance. This allows for a more nuanced representation and effective regularization. GPQ can smoothly transition from a fully random embedding to a fully trained one by adjusting a hyperparameter k (the number of Gaussian distributions). They also introduce "Unified Partitioning" as a more efficient way to partition the embedding matrix compared to the standard "Structured Partitioning."

Datasets

The paper evaluates their methods on the following machine translation datasets:

  • WMT-2014 English to French (En → Fr): A high-resource language pair.
  • WMT-2014 English to German (En → De): A medium-resource language pair.
  • IWSLT Portuguese to English (Pt → En): A low-resource language pair.

For these datasets, they used SentencePiece to create sub-word vocabularies of sizes around 32k-37k tokens.

Key Insights and Ideas

  • Random Embeddings are Surprisingly Robust: The paper reveals that NMT models, particularly Transformers, can still achieve decent performance even with completely random word embeddings. This suggests that deeper layers of the network are capable of learning semantic and syntactic information, reducing the absolute reliance on pre-trained embeddings.
  • Task-Specific Knowledge is Key, but Not All of It: While random embeddings show surprising resilience, incorporating even a small amount of task-specific knowledge from fully-trained embeddings can significantly boost performance. GPQ strikes a balance by learning from pre-trained embeddings but compressing them heavily.
  • GPQ Offers Effective Compression and Regularization: The proposed GPQ method, especially with unified partitioning, achieves substantial compression rates (up to 5.3x) with minimal or even no drop in BLEU score. It also acts as a form of regularization, sometimes even outperforming baseline models.
  • Variance Matters: Incorporating variance information in GPQ, as opposed to standard PQ, proves beneficial, especially for models with higher variance clusters, highlighting the importance of capturing more than just the centroid of clusters.

Key Results

  • Compression: GPQ achieved a 5.3x compression rate on English to German translation. In some cases, the compressed models used significantly fewer parameters (e.g., 1.25MB vs. 62.5MB for Transformer baseline on En-Fr).
  • Performance:
    • Random Word Embeddings (RWE) with a linear layer resulted in a drop of only 1-4 BLEU points on average, and in some cases, performed better than the baseline with fewer parameters.
  • GPQ (Unified) consistently outperformed other compression methods and, in some instances, achieved performance on par with or better than the Transformer baseline, even with drastically reduced embedding sizes. For example, on En-Fr, GPQ (Unified) achieved 39.77 BLEU with only 1.28MB embedding size, compared to the Transformer baseline's 39.29 BLEU with 62.5MB.
  • GPQ demonstrated effectiveness for both Transformer and LSTM-based architectures.

Limitations and Future Work

  • Limitations:
    • While RWE showed surprising resilience, it did result in a performance drop, especially for lower-resource language pairs.
  • The paper notes that for very high compression ratios, performance might degrade.
  • The effectiveness of GPQ is shown to be dependent on the choice of cluster and group sizes, requiring careful tuning.
  • Future Work:
    • The authors suggest that their work motivates future research into handling unknown tokens through optimal cluster selection.
  • The ability to approximate embedding matrices with nearly 100% integer values opens avenues for further exploration in discrete representation learning.
  • Further investigation into the trade-offs between compression, performance, and interpretability of discrete embedding spaces is encouraged.

Summary updated

Abstract

Embedding matrices are key components in neural natural language processing (NLP) models that are responsible to provide numerical representations of input tokens.\footnote{In this paper words and subwords are referred to as \textit{tokens} and the term \textit{embedding} only refers to embeddings of inputs.} In this paper, we analyze the impact and utility of such matrices in the context of neural machine translation (NMT). We show that detracting syntactic and semantic information from word embeddings and running NMT systems with random embeddings is not as damaging as it initially sounds. We also show how incorporating only a limited amount of task-specific knowledge from fully-trained embeddings can boost the performance NMT systems. Our findings demonstrate that in exchange for negligible deterioration in performance, any NMT model can be run with partially random embeddings. Working with such structures means a minimal memory requirement as there is no longer need to store large embedding tables, which is a significant gain in industrial and on-device settings. We evaluated our embeddings in translating {English} into {German} and {French} and achieved a $5.3$x compression rate. Despite having a considerably smaller architecture, our models in some cases are even able to outperform state-of-the-art baselines.
BibTeX
@inproceedings{kumar2022improving,
  author = {Krtin Kumar and Mehdi Rezagholizadeh and Yiu Sing Lau and Qun Liu},
  title = {{Improving Neural Machine Translation with Compact Word Embedding Tables}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.1609/aaai.v36i10.21340},
}
Figure