All Word Embeddings from One Embedding

Sho Takase, Sosuke Kobayashi

TL;DR

The authors propose ALONE (all word embeddings from one), a method that replaces massive word embedding matrices with a single shared base embedding and a set of memory-efficient, non-trainable filters. By passing these filtered representations through a small feed-forward network, they achieve state-of-the-art results in translation and summarization while using up to 8x fewer embedding parameters.

What Problem Does This Paper Address?

In modern Natural Language Processing (NLP), word embeddings are the bridge between discrete text and continuous vectors. However, these embeddings often constitute the largest portion of a model's parameters. For a vocabulary of size $V$ and an embedding dimension $D_e$, the matrix size is $V \times D_e$.

In a standard Transformer model for English-to-German translation, the embedding matrix can account for one-fourth of the total parameters. This "embedding bottleneck" makes models expensive to store, difficult to deploy on edge devices, and limits the memory available to scale up the actual processing layers (like the Transformer blocks themselves). Previous attempts to shrink these matrices often required complex multi-step training or additional trainable parameters to map words to compressed representations.

Methods

The authors introduce ALONE, which constructs every word's representation from a single shared base embedding $o$.

Comparison of conventional embedding lookup versus the proposed ALONE method.

The process follows three main steps:

  1. Filter Construction: Instead of storing a unique vector for every word, they use a "source matrix" (a codebook) of random, non-trainable vectors. Each word is randomly assigned a small set of these vectors, which are summed and transformed to create a word-specific filter $m_w$. Because these assignments are random and the vectors are fixed, they require almost no memory (just a random seed).
  2. Transformation: The shared base embedding $o$ is modified by the filter $m_w$ using an element-wise product ($m_w \odot o$).
  3. Expansion via FFN: To regain the expressiveness lost by sharing a base embedding, the result is passed through a small Feed-Forward Network (FFN). This FFN is trainable and learns how to project these filtered "proto-embeddings" into a high-quality vector space.

The authors tested two types of filters: Binary Masks (acting like a word-specific dropout) and Real Number Vectors (Gaussian noise).

Datasets

The researchers evaluated ALONE across three distinct tasks:

  • Word Embedding Reconstruction: Using the pre-trained GloVe (300-dimensional) embeddings as a target to see if ALONE could mimic traditional embeddings.
  • Machine Translation: The WMT 2014 English-to-German dataset (4.5M sentence pairs).
  • Summarization: The DUC 2004 task 1 (headline generation).

Key Insights and Ideas

  • Randomness is Sufficient: One of the most surprising findings is that the word-to-filter assignments don't need to be learned. Randomly assigning words to fixed "source" vectors works remarkably well, provided the FFN is there to interpret the result.
  • The Power of the FFN: The FFN acts as a universal "decoder" that takes a filtered version of the shared embedding and expands it into a distinct word representation. This shifts the burden of "memory" from a massive matrix to a small set of neural weights.
  • Parameter Efficiency: By increasing the internal dimension of the FFN ($D_{inter}$), the model can achieve higher accuracy with a negligible increase in parameters compared to the linear growth of a traditional embedding matrix.

Key Results

  • Translation: On WMT 2014, ALONE achieved a 27.61 BLEU score, slightly outperforming the conventional Transformer (27.12) while using only 8.4M embedding parameters compared to the standard 16.8M.
  • Summarization: On DUC 2004, ALONE outperformed the previous state-of-the-art (Transformer+LRPE+PE) while using only 1.0M embedding parameters—a staggering 1/8th of the original size.
  • Expressiveness: In the GloVe reconstruction task, ALONE was able to achieve a Spearman’s rank correlation nearly identical to the original embeddings, proving it can capture complex semantic relationships despite the massive compression.

Limitations and Future Work

  • Convergence Speed: The authors noted that models using Binary Masks converged significantly slower than those using real-valued filters or conventional embeddings, particularly in summarization tasks.
  • Computational Overhead: While ALONE saves memory, it adds a small amount of computation (the FFN pass) for every input token. In extremely high-throughput scenarios, this might be a trade-off to consider.
  • Scaling to Large Language Models (LLMs): The paper focuses on encoder-decoder models like the Transformer. Future work could explore how ALONE scales to massive decoder-only models (like GPT variants) where vocabulary sizes are even larger.

Summary updated

Abstract

In neural network-based models for natural language processing (NLP), the largest part of the parameters often consists of word embeddings. Conventional models prepare a large embedding matrix whose size depends on the vocabulary size. Therefore, storing these models in memory and disk storage is costly. In this study, to reduce the total number of parameters, the embeddings for all words are represented by transforming a shared embedding. The proposed method, ALONE (all word embeddings from one), constructs the embedding of a word by modifying the shared embedding with a filter vector, which is word-specific but non-trainable. Then, we input the constructed embedding into a feed-forward neural network to increase its expressiveness. Naively, the filter vectors occupy the same memory size as the conventional embedding matrix, which depends on the vocabulary size. To solve this issue, we also introduce a memory-efficient filter construction approach. We indicate our ALONE can be used as word representation sufficiently through an experiment on the reconstruction of pre-trained word embeddings. In addition, we also conduct experiments on NLP application tasks: machine translation and summarization. We combined ALONE with the current state-of-the-art encoder-decoder model, the Transformer [36], and achieved comparable scores on WMT 2014 English-to-German translation and DUC 2004 very short summarization with less parameters.
BibTeX
@inproceedings{takase2020all,
  author = {Sho Takase and Sosuke Kobayashi},
  title = {{All Word Embeddings from One Embedding}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=tcR18efkJpK},
}
Figure