From Fully Trained to Fully Random Embeddings: Improving Neural Machine Translation with Compact Word Embedding Tables

Krtin Kumar, Peyman Passban, Mehdi Rezagholizadeh, Yiu Sing Lau, Qun Liu

Introduction

In modern Neural Machine Translation (NMT), word embedding tables are often the primary drivers of memory consumption. For a standard Transformer model with a 50,000-word vocabulary, the embedding layer can account for over 36% of the total parameters. This footprint creates a significant bottleneck for deploying high-performance models on edge devices, such as smartphones and IoT hardware, or in high-throughput industrial environments. This research challenges the traditional assumption that NMT models require massive, precisely trained embedding tables, proposing that the deeper layers of a Transformer are robust enough to handle significantly compressed or even random input representations.

Rethinking Embedding Necessity

The researchers explored a fundamental question: how much semantic and syntactic information must actually be stored in the embedding layer? They investigated two primary approaches to reduce this footprint: Random Word Embeddings (RWE) and Gaussian Product Quantization (GPQ).

Random Word Embeddings (RWE)

In the RWE approach, embeddings are initialized from a standard Gaussian distribution and kept frozen throughout the training process. By never updating these parameters, the model relies entirely on its internal layers to interpret the fixed "fingerprints" assigned to each word. While this leads to a minor drop in BLEU points (1 to 4), it demonstrates that as long as each word has a unique, consistent identifier, the Transformer architecture can learn to translate effectively.

Gaussian Product Quantization (GPQ)

GPQ represents the core technical contribution of the paper. It extends standard Product Quantization (PQ)—which clusters sub-vectors to save space—by modeling each cluster as a Gaussian distribution. Instead of storing a single centroid, GPQ stores both a mean and a variance for each cluster, creating a probabilistic latent space.

The Gaussian Product Quantization (GPQ) process: extracting mean and variance from original embeddings to create a probabilistic codebook for reconstruction.

Key innovations in GPQ include:

  • Unified Partitioning: Unlike standard PQ, which partitions columns independently, Unified Partitioning exploits redundancies across the entire matrix for superior compression.
  • Knowledge Distillation: The Gaussian parameters are learned from a pre-trained "teacher" embedding matrix, effectively distilling complex semantic relationships into a compact format.

Experimental Results

The models were evaluated across three benchmarks: WMT 2014 English-French (high-resource), WMT 2014 English-German (medium-resource), and IWSLT Portuguese-English (low-resource).

Performance and Compression

Surprisingly, GPQ-compressed models often outperformed fully-trained baselines. On the English-French task, the GPQ model achieved 40.51 BLEU compared to the baseline's 39.29. The authors suggest that the compressed, discrete latent space acts as a form of regularization, preventing the model from overfitting on the embedding layer.

GPQ achieved a 5.3x compression rate for the embedding tables. The efficiency of this method is most evident when looking at the parameter counts: the researchers found that as few as 50 Gaussian distributions (represented by 100 floating-point values) were sufficient to approximate the entire embedding matrix without performance loss.

Comparison of parameter counts across quantization methods. GPQ (Unified) drastically reduces floating-point parameters to just 100 while maintaining the integer index structure.

Conclusion

This work demonstrates that NMT models can function effectively in a nearly discrete space consisting of integer indices and a handful of floating-point Gaussian parameters. By shifting the burden of semantic representation from the lookup table to the Transformer's internal layers, GPQ provides a path toward significantly more efficient NMT models that maintain, or even exceed, the translation quality of their uncompressed counterparts. Future work may extend these findings beyond NMT to other NLP tasks requiring deep semantic nuance, such as natural language inference.

Summary updated

Abstract

Embedding matrices are key components in neural natural language processing (NLP) models that are responsible to provide numerical representations of input tokens.\footnote{In this paper words and subwords are referred to as \textit{tokens} and the term \textit{embedding} only refers to embeddings of inputs.} In this paper, we analyze the impact and utility of such matrices in the context of neural machine translation (NMT). We show that detracting syntactic and semantic information from word embeddings and running NMT systems with random embeddings is not as damaging as it initially sounds. We also show how incorporating only a limited amount of task-specific knowledge from fully-trained embeddings can boost the performance NMT systems. Our findings demonstrate that in exchange for negligible deterioration in performance, any NMT model can be run with partially random embeddings. Working with such structures means a minimal memory requirement as there is no longer need to store large embedding tables, which is a significant gain in industrial and on-device settings. We evaluated our embeddings in translating {English} into {German} and {French} and achieved a $5.3$x compression rate. Despite having a considerably smaller architecture, our models in some cases are even able to outperform state-of-the-art baselines.
BibTeX
@inproceedings{kumar2022from,
  author = {Krtin Kumar and Peyman Passban and Mehdi Rezagholizadeh and Yiu Sing Lau and Qun Liu},
  title = {{From Fully Trained to Fully Random Embeddings: Improving Neural Machine Translation with Compact Word Embedding Tables}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.1609/aaai.v36i10.21340},
}
Figure